generate logs in grafana loki using nso observability-exporter

[sahooanup]

hi ,

I am trying to monitor all the nso logs in my Grafana Loki dashboard using nso Observability-exporter package. I have loaded the observability-exporter package successfully in my nso. When I run the setup.sh which is inside bservability-exporter package, I get 7 different container created. Those containers are 1. observability-exporter-jaeger, 2. observability-exporter-jaeger-collector, 3. observability-exporter-elasticsearch, 4. observability-exporter-influxdb, 5. observability-exporter-grafana, 6. observability-exporter-prometheus, 7. observability-exporter-otelcol.

Now my goal is to create  a loki container and integrate that with Grafana dashboard so that I should be able to see all the logs in my Grafana Dashboard. How to achieve this please suggest.

Thanks

[bigevilbeard]

You can use the official Loki Docker image to create a Loki container. I found an example so here is how to command to create a Loki container named observability-exporter-loki you will need to do this to create a Docker volume loki-volume to store the Loki data, and a network observability-exporter-net to allow communication between the containers fyi.

docker run -d --name observability-exporter-loki --volume=loki-volume:/mnt/loki --network=observability-exporter-net grafana/loki:2.4.0 --config.file=/mnt/loki/local-config.yaml

You will also need to create a local-config.yaml file to configure Loki. Here's an example configuration, i found on their website

auth_enabled: false
server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  log_level: info
  chunk_target_size: 1048576
  table_manager:
    retention_deletes_enabled: false
    retention_period: 168h
    retention_target_size: 10485760
    chunk_idle_period: 5m
    retention_recheck_interval: 1h
    retention_soft_deletes_enabled: false
    retention_soft_delete_period: 24h
    retention_compaction_window: 168h
    retention_compaction_retries: 3
    retention_compaction_retry_delay: 1m
    retention_compaction_max_series_per_chunk: 1000
    retention_compaction_max_age: 168h
    retention_compaction_min_size: 1048576
    retention_compaction_schedule: "0 0 * * *"
    retention_compaction_working_set: 1000000000
    retention_compaction_index_cache_size: 100000000
    retention_compaction_index_cache_ttl: 1h
    retention_compaction_index_cache_max_age: 1h
    retention_compaction_index_cache_max_entries: 1000000
    retention_compaction_index_cache_max_entry_size: 10000
    retention_compaction_index_cache_max_entry_ttl: 1h
    retention_compaction_index_cache_max_entry_ttl_age: 1h
    retention_compaction_index_cache_max_entry_ttl_size: 10000
    retention_compaction_index_cache_max_entry_ttl_size_age: 1h
    retention_compaction_index_cache_max_entry_ttl_size_age_ttl: 1h
    retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl: 1h
    retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl_ttl: 1h
    retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl_ttl_ttl: 1h
    retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl_ttl_ttl: 1h
    retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl_ttl_ttl: 1h
    retention_compaction_index_cache_max_entry_ttl_size_age_ttl_ttl_ttl_ttl: 1h
    retention_compaction_index_cache_max

[sahooanup]

I used the above local-config.yaml bit that throws error and container is exiting. We must use a proper local-config.yaml file.

[bigevilbeard]

Ok try rthis version, also create a volume for the loki data and update the store section accordingly, then also, create an overrides.yaml file to configure runtime overrides if you need.

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095
  log_level: info

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory

  chunk_idle_period: 5m
  chunk_target_size: 1048576
  max_chunk_age: 2h

store:
  boltdb:
    path: /mnt/loki/index

schema_config:
  configs:
  - from: 2020-05-15
    store: boltdb
    object_store: file
    schema: v11
    index:
      period: 24h
      prefix: index_

limits_config:
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20

runtime_config:
  file: overrides.yaml

[sahooanup]

Now I am able to create loki container with the above local-config.yaml . I am not using overrides.yaml. But when I go to my Grafana Dashboard I am not seeing any logs. I am not able to pass any queries because the Label filters section is show blank. 
PFA screen shot.

[bigevilbeard]

To add label filters, click on the Label filters button and select the labels you want to filter by. For example, if you want to filter the logs based on the level and method labels, you can select them from the list and enter the filter values.

Hope this helps.

[sahooanup]

I am not finding any options when i click on Label filters button. It shows 'no options found'. Please refer to the attached screenshot. I think loki not able to fetch the nso logs. 

[bigevilbeard]

Try to manually inspect the logs in the Loki UI or use a command-line tool like loki-cli to see if labels are present.

[sahooanup]

I am not able to access loki UI. I exposed 3100 as 50109 but not able to access the UI. please suggest whats wrong i am doing. The goal is I should be able to see Label filters option in the grafana dashboard and should be able to pass queries.
Attaching Screenshots for better clarity.

[bigevilbeard]

To troubleshoot the inaccessible Loki UI, begin by examining the container logs with docker logs -f observability-exporter-loki for error messages. Next, confirm that the port mapping in your docker-compose file or docker run command correctly exposes port 3100 and maps it to port 50109 on the host. Then, inspect the Loki configuration file (local-config.yaml) to ensure it points to the appropriate storage backend (like Elasticsearch). Finally, verify that the Loki container shares the same network as other containers (such as Grafana, Elasticsearch) and that the network configuration is accurate.

Hope this helps.