So i used GPT-5 to convert the above PRQL into its SQL equivalent by giving it links to the documentation of lnav and PRQL, it works suprisingly well,

;CREATE TABLE dashboard_service AS
WITH base AS (
  SELECT
    jget(bl.log_raw_text, '/msg')      AS message,
    jget(bl.log_raw_text, '/method')   AS method,
    jget(bl.log_raw_text, '/uri')      AS uri,
    jget(bl.log_raw_text, '/headers')  AS headers,
    bl.log_line,
    bl.log_level,
    bl.log_time
  FROM bunyan_log AS bl
),
derived AS (
  SELECT
    log_line,
    log_level,
    log_time,
    (method || ' ' || uri) AS endpoint,
    regexp_match('.?(?<fn>\[.+?\])\s?(?<msg>.+)?', message) AS message_parts,
    jget(headers, '/x-request-id') AS log_opid
  FROM base
)
SELECT
  log_line,
  log_level,
  log_time,
  log_opid,
  endpoint AS metadata_endpoint,
  jget(message_parts, '/fn')  AS metadata_span_name,
  jget(message_parts, '/msg') AS msg
FROM derived;

And now i have an lnav script that update the bunyan_log table to populate log_user_opid

;UPDATE bunyan_log AS bl
SET log_user_opid = (
  SELECT p.log_opid
  FROM dashboard_service AS p
  WHERE p.log_line = bl.log_line
)

I had also apply a custom patch to bunyan_log format using the JSON below,

{
  "$schema": "https://lnav.org/schemas/format-v1.schema.json",
  "bunyan_log": {
    "hide-extra": true,
    "opid-field": "log_opid"
  }
}

And these are enough to get the opid populated and it seems like the timeline view is correct, the only thing is, each operation does not have a description

I saw in the documentation there are description for each opid in the screenshot, how would i populate this in my case? Assuming i want to use the metadata_endpoint column from the table above. This doesn't work:

    "opid": {
      "description": {
        "api-request": {
          "format": [
            {
              "field": "metadata_endpoint"
            }
          ]
        }
      }
    }

How can the description be filled in?

Is there no way to do something like, update sqlite table using PRQL query?

PRQL is focused on analytical queries and doesn't support writing to DBs.

log_opid = json.get headers "/x-request-id",

Is this request ID part of the original log message JSON object? Are you able to patch the format file to set the opid to "headers/x-request-id"?

(Would you be able to share some of your logs? It would make it a little easier to help. You can send them to [email protected] if you don't want to post them to github. That address just goes to me.)

My lnav script uses a few sqlite functions and the PRQL playground does't really output anything useful

It looks like I log the SQL being executed in the lnav debug log:

So, if you run lnav with the -d /tmp/lnav-debug.log option, you should be able to look in that file to get the actual SQL that is run.

How can the description be filled in?

I think this is only supported when the opid is directly in the logs. There's a bunch of complicated configuration to extract the description from log messages. Here is an example:

You could also just make the description part of the opid since you're constructing it.

Is this request ID part of the original log message JSON object? Are you able to patch the format file to set the opid to "headers/x-request-id"?

Yeah, headers is a field part of the original log message JSON object, its value is another JSON. I patched the format file but setting opid to headers/x-request-id doesn't work, it doesn't get populated. I made sure to delete my sessions data too just in case.

(Would you be able to share some of your logs? It would make it a little easier to help. You can send them to [email protected] if you don't want to post them to github. That address just goes to me.)

I emailed the log sample to you, email title is Sample logs for github discussion - Visualizing in timeline mode after post processing with SQL / PRQL #1550

I think this is only supported when the opid is directly in the logs. There's a bunch of complicated configuration to extract the description from log messages.

The extractor only runs when the opid is able to be extracted by the parser? And not if its manually populated using sqlite update?

You could also just make the description part of the opid since you're constructing it.

This could work yeah.

So, if you run lnav with the -d /tmp/lnav-debug.log option, you should be able to look in that file to get the actual SQL that is run.

Thanks! I'll keep that in mind.

Unfortunately, since headers is a string under the main object and not a sub-object, lnav won't drill down further (it just doesn't have the code to do that right now). So, what you're doing now seems like the right way.

Ah that is unfortunate, So it can't be a string that happens to be a valid JSON.

Since you're already rescanning the log, I would say I should add a separate column to the log vtable for the opid description so that you can UPDATE it yourself. Then, you can construct the description however you like. I've opened #1552 to track that work. (I really should've added such a column in the first place...).

Thanks! That would be a nice QOL feature.

Another question, From the timeline view, are there any way to navigate to the logs that belongs to the opid? If you see my screenshot of the timeline view above, there are a few logs associated with the opid being displayed at the bottom panel, i can't find ways for me to navigate to the panel, choose the log and see its detail, or go back to the Log view and seeing that specific log. But if i use my cursor, i can interact with it but all i see is the Copy action (i think).

I see, thanks!

Unfortunately, I don't think there's a way to go to a particular message that is in the timeline opid details panel. That's another improvement that could be done.

It turns out I was wrong and this should be working right now. When you press Shift+Q, it should pop the view and go to the log message that is focused in the preview pane. However, there is a bug preventing you from navigating to the details pane by pressing ENTER. So, you can only select the message with the mouse right now. I'm fixing this in the top-of-tree.

I also wanted to mention that Shift+Q is generic and means "pop the view and focus on the time that matches the time in the current view". So, if you're in the HIST view, pressing Shift+Q will move to the first log message that is in the focused time bucket in the histogram. The same works for the DB view if there is a log_time column that is in ascending ordered.

There is also an opposite of Shift+Q which is Shift+A. That will push the most recently popped view onto the view stack and try to match the time. It's not working very well in the timeline case since it's not matching the opid, just the timestamp. I'm going to look into fixing that on the top-of-tree.

When you press Shift+Q, it should pop the view and go to the log message that is focused in the preview pane.

I can confirm this is working.

However, there is a bug preventing you from navigating to the details pane by pressing ENTER

Ahh i was wondering about that, all it does for me when pressing enter on something in the timeline view is just highlighting that row with purple and hiding the details panel (the one that shows all the logs related to the opid at the bottom).