Netflow Data and Gephi

[bob_the_web]

Hi Guys,
A while back you helped me get my netflow data working on Gephi, with weights. As a reminder I ended up using DL format :
<src_ip>, <dst_ip>, <number of packets>

This all works beautifully

Now I would like to add time to the picture. In netflow there is a start_time and an end_time for the flow. I would like to use this to get a timeline dynamic graph.

I don't think my DL format will do this, and I have seen some examples for spreadsheet CSV format but that does not include weight. Here is my question: what is the best data input format for my netflow data? There is a lot of it so I would prefer not to have a markup format, but would prefer a CSV format if possible.

Like this??
<src_ip>, <dst_ip>, <number of packets>, <start_time>, <end_time>

From the spreadsheet CSV example I can see "<(1,10]>" as an example of time but used with a node.

Can I do a spreadsheet like this:
Source, Target, Weight, Time Interval
<src_ip>, <dst_ip>, <number of packets>, "(start_time, end_time)"

Can I do a normal CSV format (as opposed to a spreadsheet) ?
Thanks

[eduramiba]

Hi,
Yes, Spreadsheet supports weights and dynamic attributes like Time intervals.

Your CSV file like that should be fine.

Steps for import are:

• Create new project
• Go to data laboratory
• Optionally import nodes data to nodes table, before edges
• Import spreadsheet to edges table, making sure your columns are named 'Source', 'Target' and 'Weight' (optional). Case insensitive. Also remember to select types for your columns, specially TimeInterval for your 'Time interval' column.

If you did not import nodes data, check create missing nodes option when importing edges.

Eduardo

[bob_the_web]

Excellent, thanks for the quick reply. I will try that today and report back.
Cheers

[bob_the_web]

Hi,

Well the import of the CSV file for the node table worked just fine.
I did 'import spreadsheet' and selected 'node table' set the data times and it all worked. I have a node table with all of my data, and with 'node id' added.

Problem is that I can't see how to add the 'edge' table as per your list below. When I try to repeat the above and select 'edge table' and load the spreadsheet it complains:
"Edges table needs a 'Source' and 'Target' column with node ids

The node ids were added to the node table when I did the first import, but they are not in the spreadsheet?

Here is the first two lines of the CSV file:
'source','target','weight','time interval'
11.11.11.11,22.22.22.22,1,"<(1335972406,1336577206)>"



Thanks for any help.

Hi,
Steps for import are:

• Create new project
• Go to data laboratory
• Optionally import nodes data to nodes table, before edges
• Import spreadsheet to edges table, making sure your columns are named 'Source', 'Target' and 'Weight' (optional). Case insensitive. Also remember to select types for your columns, specially TimeInterval for your 'Time interval' column.

[bob_the_web]

Is it because when I try to import the 'edge table' it is not recognising the 'source' and 'target' as node ids because they are strings?

They are unique strings (representing an IP address) but is the edge table expecting an integer?

Here is the first two lines of the CSV file:
'source','target','weight','time interval'
11.11.11.11,22.22.22.22,1,"<(1335972406,1336577206)>"

[eduramiba]

No, it accepts strings.
I guess the problem is your CSV header is:

Code: Select all

'source','target','weight','time interval'

Try to change it to:

Code: Select all

source,target,weight,time interval

In CSV you use double quotes to make sure some text is correctly recognised, but not single quotes.

Eduardo

[bob_the_web]

Yes, that was it. All working now - thanks so much!

Final question: I don't really want to use a time interval, I would rather use a 'first seen' datetime. In the example below I achieve this by converting first seen datetime to unix timestamp and then add a big time offset to make sure the node stays - creating an artificial interval.

Can I use just a first seen datetime instead of interval? If not this will do.
Thanks

No, it accepts strings.
I guess the problem is your CSV header is:

Code: Select all

'source','target','weight','time interval'

Try to change it to:

Code: Select all

source,target,weight,time interval

In CSV you use double quotes to make sure some text is correctly recognised, but not single quotes.

Eduardo

[phreakocious]

I believe you can set the end time to 'null' to make it persist forever. (this works in Java) ... Hope that helps!

[eduramiba]

Hi,
You can write this <[2.0, Infinity]> for example.

Eduardo