JQ profile

Read Updated in 18 minutes August 24, 2021

Adam Gordon Bell

Background on this page: Finger, Header and Google What is JQ Use JQ to select elements Use JQ to select arrays Use JQ To put elements into arrays Use JQ to select multiple fields Use JQ To put elements into objects A small example returns GitHub use JQ to sort and count JQ built-in functions pipes and filters use JQ Next step 1 in mastering JQ in mapping and selection review. Complete the JQ tutorial 2. Try using your memory first 3. Use it 4.

Background: fingers, head and Google

Whenever I reach the stop point of work, I use the bash alias named gwip1 to create a “work in Progress” commit. It happened without my conscious thought. Just as my fingers know about vim key bindings, they know about GWIP.

The other moves, I know how they work, but I have to think about them every time. They’re in my head, not my fingers. 2

However, some things never stay in my mind or on my fingers, and I have to Google them every time. Jq is one of them.

I know it’s a powerful tool, but I always go back to Google and copy and paste solutions from somewhere. So I solved my problem, but never learned the tool.

It is time to solve this problem. ** In this article, I delve into the basic building blocks of JQ so that you can understand how JQ works. ** Sure, you might still have to go to Google to look up function names or check your syntax from time to time, but at least you’ll have a solid foundation.

What is the JQ

Jq is a lightweight command line JSON processor. I installed it using BREW (Brew Install JQ), but it is a portable executable and therefore easy to install on Linux, Windows, or macOS. To use it, you need to build one or more filters and apply those filters to JSON documents.

The simplest filter is the identity filter, which returns all its inputs (.) :

$ echo '{"key1":{"key2":"value1"}}' | jq '.'
{
  "key1": {
    "key2": "value1"
  }
}
Copy the code

This filter is handy for nicely printing JSON documents. I’ll skip the nice print and use JQ directly to convert JSON documents.

Use JQ to select elements

I will use jQ to filter the data returned by the GitHub repository API. The default data I return is as follows:

$ curl https://api.github.com/repos/stedolan/jq
{
  "id": 5101141,
  "node_id": "MDEwOlJlcG9zaXRvcnk1MTAxMTQx",
  "name": "jq",
  "full_name": "stedolan/jq",
  "private": false,
  "owner": {
    "login": "stedolan",
    "id": 79765
  },
  "html_url": "https://github.com/stedolan/jq",
  "description": "Command-line JSON processor",
  "stargazers_count": 19967,
  "watchers_count": 19967,
  "language": "C",
  "license": {
    "key": "other",
    "name": "Other",
    "spdx_id": "NOASSERTION",
    "url": null,
    "node_id": "MDc6TGljZW5zZTA="
  }
}
Copy the code

Jq lets us treat a JSON document as an object and select its elements.

Here’s how I filter the JSON document to select the value of the name key:

$ curl https://api.github.com/repos/stedolan/jq | jq ' .name' 
"jq"
Copy the code

Also, for selecting the value of the owner key:

$ curl https://api.github.com/repos/stedolan/jq | jq ' .owner' 
{
    "login": "stedolan",
    "id": 79765
}
Copy the code

You can drill as far as possible like this:

$ curl https://api.github.com/repos/stedolan/jq | jq ' .owner.login' 
"stedolan"
Copy the code

What I learned: Object Identifiers – indexes

Jq allows you to select an element in a JSON document as if it were a JavaScript object. Just start with.(for the entire document) and drill down to the value you want. It ends up looking like this:

jq '.key.subkey.subsubkey'
Copy the code

Use JQ to select arrays

If you curl use the GitHub Issues API, you will get a list of questions:

$ curl https://api.github.com/repos/stedolan/jq/issues?per_page=5 [ { "id": 966024429, "number": 2341, "title": "Question about license.", "body": "I order like to create a [winget] (https://github.com/microsoft/winget-cli) package for jq. 🙏 🏻"}, {" id ": 962477084, "number": 2340, "title": "visibility of wiki pages", "body": "The visibility of wiki pages to search engines is generally limited;  for example, the search result for \"jq Cookbook\" looks like this:" }, { "id": 955350543, "number": 2337, "title": Release 1.6 does not have pre-autoreconf' Ed configure script ', "body": "If you have a usage question, please ask us on either Stack Overflow (https://stackoverflow.com/questions/tagged/jq) or in the #jq channel (http://irc.lc/freenode/%23jq/) on Freenode (https://webchat.freenode.net/)." }, { "id": 954792209, "number": 2336, "title": "Fix typo", "body": "" }, { "id": 940627936, "number": 2334, "title": "Compile error messages don't provide column only line number", "body": "Compile errors in filter expressions don't include the column number where the parser approximately or exactly locates the error. Most filter expressions are one-liners (are multiple lines even supported?), so the information that the error is on line 1 is not helpful." } ]Copy the code

To get a specific element in an array, give the JQ index:

$ curl https://api.github.com/repos/stedolan/jq/issues?per_page=5 | jq '.[4]' 
 {
    "id": 940627936,
    "number": 2334,
    "title": "Compile error messages don't provide column only line number",
    "body": "Compile errors in filter expressions don't include the column number where the parser approximately or exactly locates the error. Most filter expressions are one-liners (are multiple lines even supported?), so the information that the error is on line 1 is not helpful."
  }
Copy the code

Side note: Array indexjq

Array indexes have some handy syntax that is useful.

You can choose the range:

$echo [1, 2, 3, 4, 5] "" | jq '. [then] [3, 4]Copy the code

You can select a side range:

$echo [1, 2, 3, 4, 5] "" | jq '. [2:] [three, four, five]Copy the code

Additionally, you can use negation to select from the end:

$echo [1, 2, 3, 4, 5] "" | jq '. [- 2:] [4, 5)Copy the code

You can use array indexes with object indexes:

$ curl https://api.github.com/repos/stedolan/jq/issues?per_page=5 | jq '.[4].title' 
"Compile error messages don't provide column only line number"
Copy the code

You can use [] to get all the elements in the array. For example, here’s the title of how I got the question returned by the API request:

$ curl https://api.github.com/repos/stedolan/jq/issues?per_page=5 | jq '.[].title' "Question about license." "visibility Of wiki pages" "Release 1.6 does not have pre-autoreconf' Ed configure script" "Fix typo" "Compile error messages don't provide column only line number"Copy the code

What I learned: Array indexes

Jq allows you to select entire arrays [], specific elements [3], or ranges, [2:5] and combine them with object indexes as needed.

It ends up looking like this:

jq '.key[].subkey[2]
Copy the code

Side note: Remove quotes from JQ output

The -r option jq in gives you the raw string if needed.

$ echo '["1","2","3"]' | jq -r '.[]'
1
2
3
Copy the code

The output can be combined under the -j (for connections) option.

$ echo '["1","2","3"]' | jq -j '.[]'
123
Copy the code

Puts an element into an array`jq`

Once you start using array indexes to select elements, you run into a new problem. The data returned will not be a valid JSON document. In the example above, the question titles are separated by new lines:

"Undefined of wiki pages" "Release 1.6 does not have pre-autoreconf' Ed configure script" "Fix  typo" "Compile error messages don't provide column only line number" ...Copy the code

In fact, whenever you ask JQ to return a collection of unwrapped elements, it prints them on a new line. You can see this by explicitly asking JQ to ignore its input and return two numbers:

$echo '" "' | jq '1, 2, 1, 2Copy the code

You can solve this problem in the same way that 1,2 converted text to an array in JavaScript: by wrapping it in an array constructor […] .

$echo '" "' | jq '[1, 2] [1, 2]Copy the code

Similarly, to put a generated result collection into a JSON array, wrap it in an array constructor […] .

My GitHub question title filter (.[].title) then becomes [.[].title] like this:

$ curl https://api.github.com/repos/stedolan/jq/issues?per_page=5 | \ jq '[ .[].title ] ' [ "Question about license.", "Visibility of wiki pages", "Release 1.6 does not have pre-autoreconf' Ed configure script", "Fix typo", "Compile error messages don't provide column only line number" ]Copy the code

I now have a valid JSON document.

What I learned: Array constructors

If your JQ query returns more than one element, they are separated by a newline character.

$ echo '[{"a":"b"},{"a":"c"}]' | jq -r '.[].a'
  "b"
  "c"
Copy the code

To convert these values into JSON arrays, you do something similar to creating arrays in JavaScript: wrap the values in an array constructor ([…]). ).

It ends up looking like this:

$ echo '[{"a":"b"},{"a":"c"}]' | jq -r '[ .[].a ]'
[
  "b",
  "c"
]
Copy the code

use`jq`Select multiple fields

The GitHub Questions API has a lot of details that I don’t care about. I want to select several fields from the returned JSON document and leave the rest behind.

The simplest method is to specify multiple filters using:

$ curl https://api.github.com/repos/stedolan/jq/issues?per_page=2 | \ 
  jq ' .[].title, .[].number'
"Question about license."
"visibility of wiki pages"
2341
2340
Copy the code

But this returns one choice after another. To change the order, I can break down the array selector:

$ curl https://api.github.com/repos/stedolan/jq/issues?per_page=2 | \
  jq '.[] |  .title, .number'
"Question about license."
2341
"visibility of wiki pages"
2340
Copy the code

The reconstruction using a pipe (|), I will soon to talk about it, and run on each array element I object selector (. Title and. Number).

If you wrap the query in an array constructor, you get the following information:

$ curl https://api.github.com/repos/stedolan/jq/issues?per_page=2 | \
  jq '[ .[] |  .title, .number ]'
[
  "Question about license.",
  2341,
  "visibility of wiki pages",
  2340
]
Copy the code

But this is still not the JSON document I need. To put these values into the appropriate JSON object, I need an object constructor {… }.

Put elements into objects`jq`

Before showing how my GitHub query uses object constructors, let’s look at some simple examples.

A little example

I have an array containing my name ([“Adam”,”Gordon”,”Bell”]) and I want to turn it into a JSON object like this:

{
  "first_name":"Adam",
  "last_name":"Bell"
}
Copy the code

I can use an array index like this to select the elements I want:

$ echo '["Adam","Gordon","Bell"]' | jq -r '.[0], .[2]'
Adam
Bell
Copy the code

To wrap the values into the shape I want, I can replace the values with the array index that returns them:

{
  "first_name":.[0],
  "last_name":.[2]
}
Copy the code

Or on a line like this:

$ echo '["Adam","Gordon","Bell"]' | jq -r '{ "first_name":.[0], "last_name": .[2]}'
{
  "first_name": "Adam",
  "last_name": "Bell"
}
Copy the code

This syntax is the same as for creating objects in a JSON document. The only difference is that you can use the objects and array queries that you create as values.

Returns the lot

Going back to my GitHub API problem, to wrap numbers and titles into an array, I use object constructors like this:

$ curl https://api.github.com/repos/stedolan/jq/issues?per_page=2 | \ jq '[ .[] | { title: .title, number: .number} ]' [ { "title": "Question about license.", "number": 2341 }, { "title": "visibility of wiki pages", "number": 2340}]Copy the code

What I learned: Object constructors

To put the elements of your choice back into the JSON document, you wrap them in an object constructor {… }.

If you build a JSON object with several selectors, it will end up looking like this:

jq '{ "key1": <<jq filter>>, "key2": <<jq filter>> }'
Copy the code

This is the same syntax as an object in a JSON document, but jQ you can use filters as values. 4

Use JQ for sorting and counting

My next question is I want to summarize some of this JSON data. Each question returned by GitHub has a set of tags:

$ curl https://api.github.com/repos/stedolan/jq/issues/2289 | \ jq ' { title: .title, number: .number, labels: .labels} '{"title": "Bump jinja2 from 2.10 to 2.11.3 in /docs", "number": 2289, "labels": [ "feature request", "dependencies" ] }Copy the code

`jq`Built-in function

If I wanted to sort the labels alphabetically, I could use the built-in sort function. Here’s how it works:

$  echo '["3","2","1"]' | jq 'sort'
["1", "2", "3"]
Copy the code

This is similar to the way I sort arrays in JavaScript:

const l = ["3","2","1"];
l.sort();
Copy the code

This reflects the JavaScript functionality and other built-in plugins available, like Length, Reverse and toString, which can all be used in a similar way:

$  echo '["3","2","1"]' | jq 'reverse'
["1", "2", "3"]
$  echo '["3","2","1"]' | jq 'length'
3
Copy the code

If I can combine these built-in functions with the selectors I’ve built so far, I can solve my label sorting problem. So I’m going to show you that.

What I learned:jqBuilt-in plug-ins

Jq has a lot of built-in functionality. It’s probably a lot to remember, but built-in functions tend to mirror JavaScript functions, so you might be in luck to try them out before heading to the JQ manual. 5

Piping and filter

Before I can sort labels from GitHub API requests, I need to explain the pipes and filters in JQ.

Jq is a filter in the UNIX command line sense. You through the pipe (|) pass JSON documents to it, it will be to filter and its output to standard output. I can easily use this feature to link together JQ calls as follows:

echo '{"title":"JQ Select"}' | jq '.title' | jq 'length'
9
Copy the code

This is a long but simple way to determine the length of a string in a JSON document. You can use the same idea to combine the various JQ built-in functions with the functionality I’ve shown so far. But there is an easier way. You can use pipes JQ internally, conceptually they work the same as shell pipes:

echo '{"title":"JQ Select"}' | jq '.title | length'
9
Copy the code

Here are some more examples:

.title | lengthWill return the length of the title
.number | tostringReturn the problem number as a string
.[] | .keyWill returnkeyThe value of the key in the array (this is equivalent to this.[].key)

This means sorting my tag array is easy. I can be. Labels instead. Labels | sort:

$ curl https://api.github.com/repos/stedolan/jq/issues/2289 | \ jq ' { title: .title, number: .number, labels: . Labels | sort} '{" title ":" Bump jinja2 from 2.10 to 2.11.3 in/docs ", "number" : 2289, "labels" : [ "dependencies", "feature request" ] }Copy the code

If you just want a simple tag count:

$ curl https://api.github.com/repos/stedolan/jq/issues/2289 | \ jq ' { title: .title, number: .number, labels: . Labels | length} '{" title ":" Bump jinja2 from 2.10 to 2.11.3 in/docs ", "number" : 2289, "labels" : 2}Copy the code

What I learned: Pipes and filters

All the content in jq is a filter, you can be used in combination with the pipe (|). This mimics the behavior of UNIX shells.

You can use pipes and JQ built-in functions to build complex transformations from simple operations.

It ends up looking like this:

jq ' .key1.subkey2[] | sort ' # sorting
jq ' .key2.subkey | length' # length of string or array
jq ' .key3 | floor | tostring | length' # and so on
Copy the code

Use JQ mapping and selection

The list of questions I’m looking at has a lot of low quality questions. 6 Suppose I want to grab all the tagged items. This will allow me to skip all the driving to fix my problems.

Unfortunately, you can’t do this using the GitHub API unless you specify all possible labels in your query. However, I can do this by using JQ. To do so, however, I will need more JQ functionality.

So far, my query looks like this:

  jq '[ .[] | { title: .title, number: .number, labels: .labels | length } ]'
Copy the code

The first thing I can do is use map.

  jq 'map({ title: .title, number: .number, labels: .labels | length }) 
Copy the code

map(…) Let’s untangle an array, apply a filter, and rewrap the result back into an array. You can use it as a kind of shorthand, [[]] |… It comes up a lot in my experience, so it’s worth keeping in mind.

I can combine it with a SELECT statement like this:

map(select(.labels > 0))
Copy the code

Select is a built-in function that takes a Boolean expression and returns only matched elements. It is similar to a clause in a WHERESQL statement or an array filter in JavaScript.

Just like Map, I find Select comes up a lot, so while you may have to go back to this article or Google it the first few times you need it, fortunately, it will start sticking to your memory after that.

Put it all together and it looks like this:

curl https://api.github.com/repos/stedolan/jq/issues?per_page=100 | \ jq 'map({ title: .title, number: .number, labels: . Labels | | length}) map (the select (. Labels > 0)) '[{" title ":" Bump LXML from 4.3.1 to 4.6.3 in/docs ", "number" : 2295, "labels": 1}, {"title": "Bump pyyaml from 3.13 to 5.4 in /docs", "number": 2291, "Labels ": 1}, {"title": "Bump jinja2 from 2.10 to 2.11.3 in /docs", "number": 2289, "Labels ": 1}, {"title": Bump jinja2 from 2.10 to 2.11.3 in /docs", "number": 2289, "labels": 1}, {"title": "Debugging help through showing pipeline intermediates. ", "number": 2206, "labels": 1 } ]Copy the code

This uses three object indexes, two maps, two pipes, a length function, and a SELECT predicate. But if you keep following it, it all makes sense. This is just strapping the filters together until you get the desired result.

Now let’s talk about how to put this knowledge into practice.

In the review

Here’s what I learned

Here’s what I’ve learned so far:

Jq allows you to start with a. And accesses keys and arrays to select the element as if it were a JavaScript object (which it is). This functionality is created using the object and array index of the jqJSON document as follows:

jq '.key[0].subkey[2:3].subsubkey'
Copy the code

Jq programs can include object constructors {… } and array constructors […] . When you want to use the above indexes to package content extracted from a JSON document, you can use these:

jq '[ { key1: .key1, key2: .key2 }  ]'
Copy the code

Jq includes built-in function (length, sort, select, map) and pipe (|), you can combine them together, just like a combined on the command line pipe and filter:

  jq 'map({ order-of-magitude: .items | length | tostring | length }) 
Copy the code

Grasp the next steps`jq`

Reading (or writing) a tool is not enough to master it. Action is needed. This is how I consolidated this knowledge:

1. Complete`jq`The tutorial

A JQ-Tutorial is not a tutorial at all, but a collection of about 20 interactive exercises that test your knowledge of JQ. I find it very helpful.

2. Try using your memories first

Whenever I need to extract data or transform JSON documents, I try to do it first without looking anything up. If I can’t remember, sometimes jQTerm with auto-completion helps. Often, I still need to find something, but science shows that repeated searches create reservations. So over time, my retention rate should improve.

3. Use it

If you don’t use a tool, you’ll never master it. So, when I have a task that I can use to solve, jQ is what I use. At least for a while, even if there’s an easier way. JSON is everywhere, whether exploring REST apis or looking at Docker Inspect results, so opportunities abound.

4. Learn more

Finally, to deepen my knowledge, I am learning about recursive descent, declaring variables, and the advanced features in defining functions and manuals. Of course, these things rarely appear, but after I wrote them, I became fascinated with the tool.

It’s not necessary to do all of this, but if you follow some of these steps, I think using JQ will become second nature.

conclusion

So far, I’ve only covered JQ. The JQ query language is a complete programming language and you can do a lot of exciting things with it. You converted from JSON to CSV. You can define your own functions and even find prime numbers that have jq:

# Denoting the input by $n, which is assumed to be a positive integer, # eratosthenes/0 produces an array of primes less than or equal to $n: def eratosthenes: (. + 1) as $n | (($n|sqrt) / 2) as $s | [null, null, range(2; $n)] | reduce (2, 1 + (2 * range(1; $s))) as $i (.; erase($i)) | map(select(.) );Copy the code

However, simple things — like element selection, keystrokes, or value filtering — are often what you need.

I hope this helps make your JQ more approachable, and that you no longer need to go to Google 7 every time you want to query a JSON document.

While you are here:

Earthly is the grammar for defining construction. It works with your existing build system. Get repeatable and understandable builds immediately.

JQ profile

JQ profile

Background: fingers, head and Google

What is the JQ

Use JQ to select elements

Use JQ to select arrays

Puts an element into an arrayjq

usejqSelect multiple fields

Put elements into objectsjq

A little example

Returns the lot

Use JQ for sorting and counting

jqBuilt-in function

Piping and filter

Use JQ mapping and selection

In the review

Grasp the next stepsjq

1. CompletejqThe tutorial

2. Try using your memories first

3. Use it

4. Learn more

conclusion

Related Posts

Interpretation technique salon | etheric fang beige book, learning intelligent application development contract (Beijing)

Today is a summary of common python error prompts for beginners

The Cocos engine helps game developers break through

Puts an element into an array`jq`

use`jq`Select multiple fields

Put elements into objects`jq`

`jq`Built-in function

Grasp the next steps`jq`

1. Complete`jq`The tutorial