Data from Different Project

[natrlhy]

Hi,

I've been running https://github.com/liveaverage/dc-powerwall-dashboard for some time, and it also runs an InfluxDB. Would I be able to use the existing data somehow with this project? I know it's a pretty open ended question and general but I wanted to start the conversation to see how I may be able to bring that data into running this project instead.

Thanks!

[jasonacox]

Hi @natrlhy - since that project is using the same URL payloads to import the Powerwall data, there is a chance that you would only need to copy the database from "PowerwallData" to "powerwall" (what this project uses).

1. Make a backup of everything before you start! :)
2. Shell in to your existing influxdb ( docker exec -it influxdb influx) and create the new database and copy measurements into new database. I think something along these lines:

CREATE DATABASE powerwall
SELECT * INTO "powerwall"."autogen".:MEASUREMENT FROM "PowerwallData"."autogen"./.*/ GROUP BY *
SELECT * INTO "powerwall"."raw".:MEASUREMENT FROM "PowerwallData"."raw"./.*/ GROUP BY *

3. Git clone Powerwall-Dashboard and before you run setup:
4. Copy your ~/dc-powerwall-dashboard/influxdb files into ~/Powerwall-Dashboard/influxdb
5. Run setup.sh

[natrlhy]

So I've created a temp influxdb to restore the data to. my compose file:

services:
  influxdb:
    ports:
      - 8088:8086
    volumes:
      - ./influxdb/data:/var/lib/influxdb
      - ./influxdb/config:/etc/influxdb
    image: influxdb:1.8

I've run influxd restore -portable /tmp/backup_02192024 inside the container and it ran fine.
When I go outside my container I don't see any data in the folder:

user@host:~$/opt/stacks/tmp-influxdb/influxdb# du -sh *
4.0K    config
4.0K    data

I'm trying to get to the stage where you mentioned:

Copy your ~/dc-powerwall-dashboard/influxdb files into ~/Powerwall-Dashboard/influxdb

But with no data in there I don't think I'm doing something right.

[natrlhy]

Hi @natrlhy - sorry to hear about SD drama. Been there. :)

Start by looking at the setup script we use for InfluxDB here: https://github.com/jasonacox/Powerwall-Dashboard/blob/main/influxdb/influxdb.sql

I suspect you are missing at least this one:

CREATE RETENTION POLICY raw ON powerwall duration 3d replication 1

Looks like the original influx DB didn't have any other policies besides autogen:

> SHOW RETENTION POLICIES ON "PowerwallData"
name    duration shardGroupDuration replicaN default
----    -------- ------------------ -------- -------
autogen 0s       168h0m0s           1        true
> SHOW RETENTION POLICIES ON "powerwall"
name    duration shardGroupDuration replicaN default
----    -------- ------------------ -------- -------
autogen 0s       168h0m0s           1        true
raw     72h0m0s  24h0m0s            1        false
strings 0s       168h0m0s           1        false
pwtemps 0s       168h0m0s           1        false
vitals  0s       168h0m0s           1        false
kwh     0s       168h0m0s           1        false
daily   0s       168h0m0s           1        false
monthly 0s       168h0m0s           1        false
grid    0s       168h0m0s           1        false
pod     0s       168h0m0s           1        false
alerts  0s       168h0m0s           1        false

[jasonacox]

If you see the data inside the container, but it is now showing up outside, that sounds like the volume mounts are not set up correctly in your compose file. You can run this to find out what it thinks it is doing:

docker inspect influxdb | grep -A 20 Mounts

[natrlhy]

Ok! I was looking through your code and also posted a question on reddit about why I can't see the data/wal/meta folders outside the contianer and it turns out one has to use a config file for influxdb. The link to my reddit post has the info but I'll put it here as well.

compose file to restore the old data to:

services:
  influxdb:
    image: influxdb:1.8
    container_name: test-influxdb
    restart: unless-stopped
    volumes:
      - ./influxdb.conf:/etc/influxdb/influxdb.conf
      - ./influxdb:/var/lib/influxdb
    ports:
      - 8088:8086

influxdb.conf file:

reporting-disabled = false
bind-address = "127.0.0.1:8088"

[meta]
  dir = "/var/lib/influxdb/meta"
  retention-autocreate = true
  logging-enabled = true

[data]
  dir = "/var/lib/influxdb/data"
  index-version = "inmem"
  wal-dir = "/var/lib/influxdb/wal"
  wal-fsync-delay = "0s"

Now the next question :)

I've been running your project for about a week now and I want to retain the data from it.
I wonder if this will work:

• In the current project with the new data run influxd backup -portable <backup_path>
• Copy that outside the container
• Follow your steps here Data from Different Project #128 (comment)
• bash into the container after running setup.sh and restore with influxd restore -portable <backup_path>

Thoughts? 😅

[jasonacox]

That seems reasonable to me. I confess, I'm not an influxdb expert, and definitely haven't moved data from other projects into mine, like you are suggesting. If I were you, I would probably stand up a separate instance to try it first. :)