jq

| jq -j '.ts |= strftime("%Y-%m-%d %H:%M:%S") | \
.request.remote_addr |= .[:-6] | \
.ts, "|", .request.remote_addr, "|" , .request.uri, "|" ,\
.request.method,"|", .request.proto,"|", .status,"|", \
.request.headers."User-Agent"[]+"\n"'\
/var/log/caddy/access.log

| jq -r 'leaf_paths | join(".")' # print schema
| jq -c '.[]|keys' |jq -s 'add' |pbcopy
| jq -r '.[].KeyValuePairs|keys' |jq -s 'add' |pbcopy

# search for any and remove nulls
| jq -c '..|.cip?|select( . != null )'

cat 2025-03-11-cloud-trail-dev-run.json \
| jq -c '.Records|reverse[]|select(.eventName != "UpdateAutoScalingGroup")|[.eventTime,.eventName,.responseElements]'

| jq -c '.Records|reverse[]|select(.eventName|IN("UpdateAutoScalingGroup","StartQuery","StopQuery")|not)|[.eventTime,.eventName,.responseElements]'

https://jqlang.github.io/jq/manual/#invoking-jq

Cookbook
- https://github.com/bcicen/jstream
- https://github.com/pkoppstein/jm

jq Guide from Baeldung

Golang version

https://remysharp.com/drafts/jq-recipes — complex transforms

https://jqterm.com

https://jqplay.org/

https://github.com/bazaarvoice/jolt — alternative for transforms?

Examples

cat FormDataCollections.json \
|jq '.[0]|. * (.KeyValuePairs|fromjson)|del(.KeyValuePairs)' \
|t

cat FormDataCollections.json \
|jq -r '.[]|. * (.KeyValuePairs|fromjson)|del(.KeyValuePairs)' \
|jq -s \
|jq -r '(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv' \
|t

# converting strings in values to json
cat actionplans.json |jq 'map_values(.|fromjson)'

Old notes

https://stedolan.github.io/jq

https://stedolan.github.io/jq/manual/

cat file |jq 'keys' # to learn structure
cat file |jq '.items[0]|keys' # to learn structure

jq '{ user: .username, email: (.attributes[] | select(.name == "email") |.value)}'

cat mro.users |jq '{ user: .username, email: (.attributes[] | select(.name == "email") |.value), status: .user_status}'|jq '. | select(.status != "FORCE_CHANGE_PASSWORD")'

cat mro.users |jq -c '. | select(.user_status != "FORCE_CHANGE_PASSWORD")'|jq -c '{ user: .username, email: (.attributes[] | select(.name == "email") |.value), status: .user_status}'

npm search aws -json |jq '[.[] |{date,name,version}]'|jq 'sort_by(.date)|.[]'

# convert splunk output
cat from-splunk.json |jq '.result._raw|fromjson' |less

# convert embedded file content to string
jq -r '.some.embedded.field.with.newlines.escaped'

# escape and capture content for embedding into json field
cat mro.txt |jq -sR |pbcopy

Sorting

StackExchange how to sort jq

# sort for compare
cat audit-prod.json| jq --sort-keys

jq '[.data[] | {id: .id, date: .date}] | sort_by(.date)' file.json

merge json (see the add command or +)
- https://stackoverflow.com/questions/19529688/how-to-merge-2-json-objects-from-2-files-using-jq

https://www.baeldung.com/linux/jq-command-json 6. Transform

{
  "query": {
    "pages": [
      {
        "21721040": {
          "pageid": 21721040,
          "ns": 0,
          "title": "Stack Overflow",
          "extract": "Some interesting text about Stack Overflow"
        }
      },
      {
        "21721041": {
          "pageid": 21721041,
          "ns": 0,
          "title": "Baeldung",
          "extract": "A great place to learn about Java"
        }
      }
    ]
  }
}

jq '.query.pages \
| [.[] | map(.) | .[] \
| {page_title: .title, page_description: .extract}\
]' wikipedia.json