Thanks to visit codestin.com
Credit goes to github.com

Skip to content
This repository was archived by the owner on Oct 29, 2024. It is now read-only.

Drop pandas DataFrame NaN values for json protocol #587

Closed
wants to merge 29 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
75d7fe8
Add _to_dict_drop_na function and use it in _convert_dataframe_to_json
tux-00 May 17, 2018
8c20ea3
Satisfy flake8
tux-00 May 18, 2018
119844a
Add dropna on tag columns
tux-00 May 18, 2018
2c15587
Change to_dict orient param to original value
tux-00 May 18, 2018
36fc27a
Merge branch 'master' into master
tux-00 Jun 27, 2018
61d6b89
Merge branch 'master' into master
xginn8 Jul 1, 2018
fe9203b
Add nan values insertion test
tux-00 Jul 29, 2018
25272aa
Add numpy to test requirements
tux-00 Jul 29, 2018
188269c
Revert last commit
tux-00 Jul 29, 2018
ecb3e18
Merge remote-tracking branch 'upstream/master'
tux-00 Aug 22, 2018
3207216
Remove numpy import
tux-00 Aug 22, 2018
112a9ab
Add trick to remove numpy dependency
tux-00 Aug 22, 2018
eaf818d
5.2.1 release
aviau Dec 7, 2018
5ad2188
release 5.2.1 in changelog
aviau Dec 7, 2018
ec86d39
Unpin setuptools to fix travis (#674)
xginn8 Feb 12, 2019
fbfca02
unpin pypy (#682)
clslgrnc Mar 14, 2019
f3f890c
Rename all mixedCase globals to snake case to appease N816 (#689)
xginn8 Mar 14, 2019
2d17360
Fixup small test docstring typo
xginn8 Mar 14, 2019
5147aed
Fix tz localize (#684)
clslgrnc Mar 14, 2019
f93010b
Bump version to 5.2.2
xginn8 Mar 14, 2019
9fd9d6e
[WIP] add py37 and recent influxdb (#692)
clslgrnc Mar 15, 2019
e5d2589
Python and influxdb supported versions (#693)
clslgrnc Mar 16, 2019
170d47e
Add CODEOWNERS file for automatic reviewers on GitHub
xginn8 Mar 20, 2019
a3d28f2
Parameter binding for client's `query()` method (#678)
clslgrnc Mar 20, 2019
9e876dc
Add CQs management methods to the client (#681)
lukaszdudek-silvair Apr 1, 2019
6f356f0
Fix a warning under Python 3.7 (#697)
Hanaasagi Apr 7, 2019
15d8231
Update delete_series docstring to differentiate between drop_database…
xginn8 Apr 8, 2019
c24d828
add consistency parameter to write_points (#664)
RonRothman Apr 8, 2019
a0cd987
Add Example for sending information to DB via UDP (#648)
shantanoo-desai Apr 10, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 33 additions & 7 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,16 @@ python:
- "2.7"
- "3.5"
- "3.6"
- "pypy-5.3.1"
- "pypy"
- "pypy3"

env:
- INFLUXDB_VER=1.2.4
- INFLUXDB_VER=1.3.9
- INFLUXDB_VER=1.4.2
- INFLUXDB_VER=1.5.4
- INFLUXDB_VER=1.2.4 # 2017-05-08
- INFLUXDB_VER=1.3.9 # 2018-01-19
- INFLUXDB_VER=1.4.3 # 2018-01-30
- INFLUXDB_VER=1.5.4 # 2018-06-22
- INFLUXDB_VER=1.6.4 # 2018-10-24
- INFLUXDB_VER=1.7.4 # 2019-02-14

addons:
apt:
Expand All @@ -20,7 +22,31 @@ addons:

matrix:
include:
- python: 2.7
- python: 3.7
dist: xenial
sudo: true
env: INFLUXDB_VER=1.2.4
- python: 3.7
dist: xenial
sudo: true
env: INFLUXDB_VER=1.3.9
- python: 3.7
dist: xenial
sudo: true
env: INFLUXDB_VER=1.4.3
- python: 3.7
dist: xenial
sudo: true
env: INFLUXDB_VER=1.5.4
- python: 3.7
dist: xenial
sudo: true
env: INFLUXDB_VER=1.6.4
- python: 3.7
dist: xenial
sudo: true
env: INFLUXDB_VER=1.7.4
- python: 3.6
env: TOX_ENV=pep257
- python: 3.6
env: TOX_ENV=docs
Expand All @@ -31,7 +57,7 @@ matrix:

install:
- pip install tox-travis
- pip install setuptools==20.6.6
- pip install setuptools
- pip install coveralls
- mkdir -p "influxdb_install/${INFLUXDB_VER}"
- if [ -n "${INFLUXDB_VER}" ] ; then wget "https://dl.influxdata.com/influxdb/releases/influxdb_${INFLUXDB_VER}_amd64.deb" ; fi
Expand Down
21 changes: 21 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,27 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/)
and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html).

## [Unreleased]

### Added
- Add `get_list_continuous_queries`, `drop_continuous_query`, and `create_continuous_query` management methods for
continuous queries (#681 thx @lukaszdudek-silvair)
- query() now accepts a bind_params argument for parameter binding (#678 thx @clslgrnc)

### Changed
- Add consistency param to InfluxDBClient.write_points (#643 thx @RonRothman)
- Update test suite to add support for Python 3.7 and InfluxDB v1.6.4 and 1.7.4 (#692 thx @clslgrnc)
- Update classifiers tuple to list in setup.py (#697 thx @Hanaasagi)
- Update documentation for empty `delete_series` confusion

### Removed

## [v5.2.2] - 2019-03-14
### Added

### Changed
- Fix 'TypeError: Already tz-aware' introduced with recent versions of Panda (#671, #676, thx @f4bsch @clslgrnc)

## [v5.2.1] - 2018-12-07
### Added

### Changed
Expand Down
1 change: 1 addition & 0 deletions CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
* @aviau @xginn8 @sebito91
4 changes: 2 additions & 2 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ InfluxDB is an open-source distributed time series database, find more about Inf
InfluxDB pre v1.1.0 users
-------------------------

This module is tested with InfluxDB versions: v1.2.4, v1.3.9, v1.4.2, and v1.5.4.
This module is tested with InfluxDB versions: v1.2.4, v1.3.9, v1.4.3, v1.5.4, v1.6.4, and 1.7.4.

Those users still on InfluxDB v0.8.x users may still use the legacy client by importing ``from influxdb.influxdb08 import InfluxDBClient``.

Expand All @@ -59,7 +59,7 @@ On Debian/Ubuntu, you can install it with this command::
Dependencies
------------

The influxdb-python distribution is supported and tested on Python 2.7, 3.5, 3.6, PyPy and PyPy3.
The influxdb-python distribution is supported and tested on Python 2.7, 3.5, 3.6, 3.7, PyPy and PyPy3.

**Note:** Python <3.5 are currently untested. See ``.travis.yml``.

Expand Down
6 changes: 6 additions & 0 deletions docs/source/examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,9 @@ Tutorials - SeriesHelper

.. literalinclude:: ../../examples/tutorial_serieshelper.py
:language: python

Tutorials - UDP
===============

.. literalinclude:: ../../examples/tutorial_udp.py
:language: python
9 changes: 8 additions & 1 deletion examples/tutorial.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@ def main(host='localhost', port=8086):
dbname = 'example'
dbuser = 'smly'
dbuser_password = 'my_secret_password'
query = 'select value from cpu_load_short;'
query = 'select Float_value from cpu_load_short;'
query_where = 'select Int_value from cpu_load_short where host=$host;'
bind_params = {'host': 'server01'}
json_body = [
{
"measurement": "cpu_load_short",
Expand Down Expand Up @@ -50,6 +52,11 @@ def main(host='localhost', port=8086):

print("Result: {0}".format(result))

print("Querying data: " + query_where)
result = client.query(query_where, bind_params=bind_params)

print("Result: {0}".format(result))

print("Switch user: " + user)
client.switch_user(user, password)

Expand Down
66 changes: 66 additions & 0 deletions examples/tutorial_udp.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
# -*- coding: utf-8 -*-
"""Example for sending batch information to InfluxDB via UDP."""

"""
INFO: In order to use UDP, one should enable the UDP service from the
`influxdb.conf` under section
[[udp]]
enabled = true
bind-address = ":8089" # port number for sending data via UDP
database = "udp1" # name of database to be stored
[[udp]]
enabled = true
bind-address = ":8090"
database = "udp2"
"""


import argparse

from influxdb import InfluxDBClient


def main(uport):
"""Instantiate connection to the InfluxDB."""
# NOTE: structure of the UDP packet is different than that of information
# sent via HTTP
json_body = {
"tags": {
"host": "server01",
"region": "us-west"
},
"time": "2009-11-10T23:00:00Z",
"points": [{
"measurement": "cpu_load_short",
"fields": {
"value": 0.64
}
},
{
"measurement": "cpu_load_short",
"fields": {
"value": 0.67
}
}]
}

# make `use_udp` True and add `udp_port` number from `influxdb.conf` file
# no need to mention the database name since it is already configured
client = InfluxDBClient(use_udp=True, udp_port=uport)

# Instead of `write_points` use `send_packet`
client.send_packet(json_body)


def parse_args():
"""Parse the args."""
parser = argparse.ArgumentParser(
description='example code to play with InfluxDB along with UDP Port')
parser.add_argument('--uport', type=int, required=True,
help=' UDP port of InfluxDB')
return parser.parse_args()


if __name__ == '__main__':
args = parse_args()
main(uport=args.uport)
2 changes: 1 addition & 1 deletion influxdb/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,4 +18,4 @@
]


__version__ = '5.2.0'
__version__ = '5.2.2'
33 changes: 27 additions & 6 deletions influxdb/_dataframe_client.py
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@ def write_points(self,
def query(self,
query,
params=None,
bind_params=None,
epoch=None,
expected_response_code=200,
database=None,
Expand All @@ -153,8 +154,18 @@ def query(self,
"""
Query data into a DataFrame.

.. danger::
In order to avoid injection vulnerabilities (similar to `SQL
injection <https://www.owasp.org/index.php/SQL_Injection>`_
vulnerabilities), do not directly include untrusted data into the
``query`` parameter, use ``bind_params`` instead.

:param query: the actual query string
:param params: additional parameters for the request, defaults to {}
:param bind_params: bind parameters for the query:
any variable in the query written as ``'$var_name'`` will be
replaced with ``bind_params['var_name']``. Only works in the
``WHERE`` clause and takes precedence over ``params['params']``
:param epoch: response timestamps to be in epoch format either 'h',
'm', 's', 'ms', 'u', or 'ns',defaults to `None` which is
RFC3339 UTC format with nanosecond precision
Expand All @@ -172,6 +183,7 @@ def query(self,
:rtype: :class:`~.ResultSet`
"""
query_args = dict(params=params,
bind_params=bind_params,
epoch=epoch,
expected_response_code=expected_response_code,
raise_errors=raise_errors,
Expand Down Expand Up @@ -202,7 +214,8 @@ def _to_dataframe(self, rs, dropna=True):
df = pd.DataFrame(data)
df.time = pd.to_datetime(df.time)
df.set_index('time', inplace=True)
df.index = df.index.tz_localize('UTC')
if df.index.tzinfo is None:
df.index = df.index.tz_localize('UTC')
df.index.name = None
result[key].append(df)
for key, data in result.items():
Expand All @@ -213,8 +226,8 @@ def _to_dataframe(self, rs, dropna=True):

return result

@staticmethod
def _convert_dataframe_to_json(dataframe,
def _convert_dataframe_to_json(self,
dataframe,
measurement,
tags=None,
tag_columns=None,
Expand Down Expand Up @@ -257,14 +270,16 @@ def _convert_dataframe_to_json(dataframe,
"h": 1e9 * 3600,
}.get(time_precision, 1)

dataframe_zip = zip(dataframe.index,
self._todict_dropna(dataframe[tag_columns]),
self._todict_dropna(dataframe[field_columns]))

points = [
{'measurement': measurement,
'tags': dict(list(tag.items()) + list(tags.items())),
'fields': rec,
'time': np.int64(ts.value / precision_factor)}
for ts, tag, rec in zip(dataframe.index,
dataframe[tag_columns].to_dict('record'),
dataframe[field_columns].to_dict('record'))
for ts, tag, rec in dataframe_zip
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got an error when all fields contain NaN values. I changed this to
for ts, tag, rec in dataframe_zip if any(rec)
to skip the points with all empty fields.

]

return points
Expand Down Expand Up @@ -386,6 +401,12 @@ def _convert_dataframe_to_lines(self,
points = (measurement + tags + ' ' + fields + ' ' + time).tolist()
return points

@staticmethod
def _todict_dropna(dataframe):
return [{k: v for k, v in m.items()
if pd.notnull(v)}
for m in dataframe.to_dict('record')]

@staticmethod
def _stringify_dataframe(dframe, numeric_precision, datatype='field'):

Expand Down
Loading