Merge pull request #1 from lionsoul2014/master

合并远程
This commit is contained in:
mohong 2017-05-25 14:51:46 +08:00 committed by GitHub
commit e4f54054c4
17 changed files with 28315 additions and 15550 deletions

1
.gitignore vendored
View File

@ -6,6 +6,7 @@
META-INF/
# Binary Files #
*.jar
!dbMaker-*.jar
# Package Files #
.settings/
.classpath

View File

@ -1,3 +1,34 @@
### 1.2.4
1, 花了近两周的时间重写了数据的升级算法,再次提高准确率,升级过程如下(此处应该有掌声):
```shell
1. ip预分段利于分布式同步升级
2. 分段同步升级,目前使用四个机器同时升级
3自动验证数据查缺补全
4数据合并
5数据格式标准化目前2个工作
1)ipip.net的数据和淘宝数据保持区域名称统一ipip.net的数据不带“省”和“市”关键字。
2)香港澳门台湾等国家信息修复淘宝原始数据有问题reported at http://git.oschina.net/lionsoul/ip2region/issues/21
6重复ip段的合并得到data目录下的ip.merge.txt
7生产data目录下的ip2region.db二进制数据库文件
```
2修复原始数据关于“香港台湾澳门”国家信息错误的bugreported at http://git.oschina.net/lionsoul/ip2region/issues/21
3将数据升级至2017/03/15同步版本
4增加c_mmap查询客户端C客户端查询文件读取使用内存映射加速感谢[Leo Ma](http://git.oschina.net/begeekmyfriend)的贡献
备注部分阿里云数据有问题缺失的部分使用免费开放的ipip.net的数据代替了。
### 1.2.3
1. 优化升级算法进行更深度的分裂可以使准确率再度提高同时数据的更新也花费了2天多的时间
2. 数据升级至2016/12/14版本的数据包括原始数据和binary数据库的数据
### 1.2.2
1. ip数据文件更新至2016/11/02版本
2. 增加ip2region.db数据库文件的java生成工具请参考README获取使用方式
3. 新增golang查询客户端的实现感谢 [@mohong122](https://github.com/mohong122) 的关注和贡献
### 1.2
1. 新增分列式升级算法大大的增加了数据的准确率基本避免之前各大网友反馈的些许ip定位错误

View File

@ -201,9 +201,9 @@
limitations under the License.
==========================================================================
The following license applies to the JQuery JavaScript library
The following license applies to the ip2region library
--------------------------------------------------------------------------
Copyright (c) 2010 John Resig, http://jquery.com/
Copyright (c) 2015 Lionsoul<chenxin619315@gmail.com>
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
@ -223,4 +223,3 @@ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

View File

@ -18,7 +18,7 @@ ip2region - 最自由的ip地址查询库ip到地区的映射库提供Bina
**4. 多查询客户端的支持0.0x毫秒级别的查询**
已经集成的客户端有java, php, c, pythonnodejsphp扩展(支持linux, php5, php7版本已支持)。
已经集成的客户端有java, php, c, pythonnodejsphp扩展(支持linux, php5, php7版本已支持)golang
提供了两种查询算法,响应时间如下:
客户端/binary算法/b-tree算法/Memory算法
@ -64,3 +64,29 @@ python:
输入ip地址开始测试第一次会稍微有点慢在运行命令后面接入binary,memory来尝试其他算法建议使用b-tree算法速度和并发需求的可以使用memory算法。
具体集成请参考不同客户端的测试源码。
**6. 如何生成ip2region.db文件**
从ip2region 1.2.2版本开始里面提交了一个dbMaker-{version}.jar的可以执行jar文件用它来完成这个工作
* 1, 确保你安装好了java环境不玩Java的童鞋就自己谷歌找找拉临时用一用几分钟的事情
* 2, cd到ip2region的根目录然后运行如下命令
```shell
java -jar dbMaker-{version}.jar -src 文本数据文件 -region 地域csv文件 [-dst 生成的ip2region.db文件的目录]
# 文本数据文件db文件的原始文本数据文件路径自带的ip2region.db文件就是/data/ip.merge.txt生成而来的你可以换成自己的或者更改/data/ip.merge.txt重新生成
# 地域csv文件该文件目的是方便配置ip2region进行数据关系的存储得到的数据包含一个city_id这个直接使用/data/origin/global_region.csv文件即可
# ip2region.db文件的目录是可选参数没有指定的话会在当前目录生成一份./data/ip2region.db文件
```
* 3, 获取生成的ip2region.db文件覆盖原来的ip2region.db文件即可
* 4, 默认的ip2region.db文件生成命令:
```shell
cd ip2region项目根目录
java -jar dbMaker-1.2.2.jar -src ./data/ip.merge.txt -region ./data/global_region.csv
# 会看到一大片的输出
```
* 5, 数据库文件的结构和原理请阅读 @冬芽 的blog[“ip2region数据库文件的结构和原理”](http://dongyado.com/tools/2016/08/18/structure-of-ip2region-database-file/)

328
binding/c_mmap/ip2region.c Normal file
View File

@ -0,0 +1,328 @@
/**
* ip2region implementation with mmap support created by Leo Ma
*
* @see #ip2region.h
* @author chenxin<chenxin619315@gmail.com>
* @author Leo Ma<http://git.oschina.net/begeekmyfriend>
* @date 2017/01/12
*/
#include "ip2region.h"
#include <unistd.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/mman.h>
/**
* create a new ip2region object
*
* @param dbFile path
*/
IP2R_API uint_t ip2region_create(ip2region_t ip2rObj, char *dbFile)
{
struct stat st;
memset(ip2rObj, 0x00, sizeof(ip2region_entry));
ip2rObj->dbFile = IP2R_MALLOC(1024);
strcpy(ip2rObj->dbFile, dbFile);
int fd = open(dbFile, O_RDONLY);
if ( fd < 0 ) {
//fprintf(stderr, "Fail to open the db file %s\n", ip2rObj->dbFile);
return 0;
}
if ( fstat(fd, &st) < 0 ) {
//fprintf(stderr, "Fail to stat the db file %s\n", ip2rObj->dbFile);
return 0;
}
ip2rObj->dbSize = st.st_size;
ip2rObj->dbBinStr = mmap(NULL, ip2rObj->dbSize, PROT_READ, MAP_PRIVATE, fd, 0);
if ( ip2rObj->dbBinStr == (void *) -1 ) {
//fprintf(stderr, "Fail to map the db file %s\n", ip2rObj->dbFile);
return 0;
}
// Close db file as soon as possible.
close(fd);
ip2rObj->headerLen = 0;
ip2rObj->HeaderSip = (uint_t *) IP2R_MALLOC(TOTAL_HEADER_LENGTH);
if ( ip2rObj->HeaderSip == NULL ) {
return 0;
}
ip2rObj->HeaderPtr = (uint_t *) IP2R_MALLOC(TOTAL_HEADER_LENGTH);
if ( ip2rObj->HeaderPtr == NULL ) {
IP2R_FREE(ip2rObj->HeaderSip);
return 0;
}
ip2rObj->firstIndexPtr = getUnsignedInt(ip2rObj->dbBinStr, 0);
ip2rObj->lastIndexPtr = getUnsignedInt(ip2rObj->dbBinStr, 4);
ip2rObj->totalBlocks = (ip2rObj->lastIndexPtr-ip2rObj->firstIndexPtr)/INDEX_BLOCK_LENGTH + 1;
return 1;
}
/**
* destroy the specifield ip2region object
*
* @param ip2region_t
*/
IP2R_API uint_t ip2region_destroy(ip2region_t ip2rObj)
{
IP2R_FREE(ip2rObj->dbFile);
IP2R_FREE(ip2rObj->HeaderSip);
IP2R_FREE(ip2rObj->HeaderPtr);
//free the db binary string
if ( ip2rObj->dbBinStr != NULL ) {
munmap(ip2rObj->dbBinStr, ip2rObj->dbSize);
}
memset(ip2rObj, 0x00, sizeof(ip2region_entry));
return 1;
}
/**
* get the region associated with the specified ip address with the memory binary search algorithm
*
* @param ip2rObj
* @param ip
* @param datablock
*/
IP2R_API uint_t ip2region_binary_search(ip2region_t ip2rObj, uint_t ip, datablock_t datablock)
{
int l, h, m, p;
uint_t sip, eip, dptr;
int dataLen, dataptr;
char *buffer;
l = 0; h = ip2rObj->totalBlocks; dptr = 0;
while ( l <= h ) {
m = (l + h) >> 1;
p = ip2rObj->firstIndexPtr + m * INDEX_BLOCK_LENGTH;
buffer = ip2rObj->dbBinStr + p;
sip = getUnsignedInt(buffer, 0);
if ( ip < sip ) {
h = m - 1;
} else {
eip = getUnsignedInt(buffer, 4);
if ( ip > eip ) {
l = m + 1;
} else {
dptr = getUnsignedInt(buffer, 8);
break;
}
}
}
if ( dptr == 0 ) return 0;
//get the data
dataLen = ((dptr >> 24) & 0xFF);
dataptr = (dptr & 0x00FFFFFF);
buffer = ip2rObj->dbBinStr + dataptr;
//fill the data to the datablock
datablock->city_id = getUnsignedInt(buffer, 0);
dataLen -= 4; //reduce the length of the city_id
memcpy(datablock->region, buffer + 4, dataLen);
datablock->region[dataLen] = '\0';
return 1;
}
IP2R_API uint_t ip2region_binary_search_string(ip2region_t ip2rObj, char *ip, datablock_t datablock)
{
return ip2region_binary_search(ip2rObj, ip2long(ip), datablock);
}
/**
* get the region associated with the specifield ip address with b-tree algorithm
*
* @param ip2rObj
* @param ip
* @param datablock
* @return uint_t
*/
IP2R_API uint_t ip2region_btree_search(ip2region_t ip2rObj, uint_t ip, datablock_t datablock)
{
int i, j;
int l, m, h, p, sptr, eptr, indexBlockLen, dataLen, dataptr;
uint_t sip, eip, idxptr, dptr;
char *buffer;
if ( ip2rObj->headerLen == 0 ) {
buffer = ip2rObj->dbBinStr + 8;
for ( i = j = 0; i < TOTAL_HEADER_LENGTH; i += 8, j++ ) {
sip = getUnsignedInt(buffer, i);
idxptr = getUnsignedInt(buffer, i + 4);
if ( idxptr == 0 ) break;
ip2rObj->HeaderSip[j] = sip;
ip2rObj->HeaderPtr[j] = idxptr;
}
ip2rObj->headerLen = j;
}
//search the header block to define the index block
l = 0; h = ip2rObj->headerLen; sptr = 0; eptr = 0;
while ( l <= h ) {
m = ((l + h) >> 1);
//perfetc matched, just return it
if ( ip == ip2rObj->HeaderSip[m] ) {
if ( m > 0 ) {
sptr = ip2rObj->HeaderPtr[m-1];
eptr = ip2rObj->HeaderPtr[m ];
} else {
sptr = ip2rObj->HeaderPtr[m ];
eptr = ip2rObj->HeaderPtr[m+1];
}
break;
}
//less then the middle value
if ( ip < ip2rObj->HeaderSip[m] ) {
if ( m == 0 ) {
sptr = ip2rObj->HeaderPtr[m ];
eptr = ip2rObj->HeaderPtr[m+1];
break;
} else if ( ip > ip2rObj->HeaderSip[m-1] ) {
sptr = ip2rObj->HeaderPtr[m-1];
eptr = ip2rObj->HeaderPtr[m ];
break;
}
h = m - 1;
} else {
if ( m == ip2rObj->headerLen - 1 ) {
sptr = ip2rObj->HeaderPtr[m-1];
eptr = ip2rObj->HeaderPtr[m ];
break;
} else if ( ip <= ip2rObj->HeaderSip[m+1] ) {
sptr = ip2rObj->HeaderPtr[m ];
eptr = ip2rObj->HeaderPtr[m+1];
break;
}
l = m + 1;
}
}
//not matched just stop it
if ( sptr == 0 ) return 0;
indexBlockLen = eptr - sptr;
buffer = ip2rObj->dbBinStr + sptr;
dptr = 0; l = 0; h = indexBlockLen / INDEX_BLOCK_LENGTH;
while ( l <= h ) {
m = ((l + h) >> 1);
p = m * INDEX_BLOCK_LENGTH;
sip = getUnsignedInt(buffer, p);
if ( ip < sip ) {
h = m - 1;
} else {
eip = getUnsignedInt(buffer, p + 4);
if ( ip > eip ) {
l = m + 1;
} else {
dptr = getUnsignedInt(buffer, p + 8);
break;
}
}
}
if ( dptr == 0 ) return 0;
dataLen = ((dptr >> 24) & 0xFF);
dataptr = (dptr & 0x00FFFFFF);
buffer = ip2rObj->dbBinStr + dataptr;
datablock->city_id = getUnsignedInt(buffer, 0);
dataLen -= 4;
memcpy(datablock->region, buffer + 4, dataLen);
datablock->region[dataLen] = '\0';
return 1;
}
IP2R_API uint_t ip2region_btree_search_string(ip2region_t ip2rObj, char *ip, datablock_t datablock)
{
return ip2region_btree_search(ip2rObj, ip2long(ip), datablock);
}
/**
* get a unsinged long(4bytes) from a specifield buffer start from the specifield offset
*
* @param buffer
* @param offset
* @return uint_t
*/
IP2R_API uint_t getUnsignedInt(char *buffer, int offset)
{
return (
((buffer[offset ]) & 0x000000FF) |
((buffer[offset+1] << 8) & 0x0000FF00) |
((buffer[offset+2] << 16) & 0x00FF0000) |
((buffer[offset+3] << 24) & 0xFF000000)
);
}
/**
* string ip to long
*
* @param ip
* @return uint_t
*/
IP2R_API uint_t ip2long(char *ip)
{
int i = 0, p = 24;
char buffer[4], *cs = ip;
uint_t ipval = 0;
while ( *cs != '\0' ) {
if ( *cs == '.' ) {
//single part length limit
if ( i > 3 ) {
ipval = 0;
break;
}
if ( p < 0 ) break;
buffer[i] = '\0';
ipval |= (atoi(buffer) << p);
p -= 8;
i = 0;
} else {
buffer[i++] = *cs;
}
cs++;
}
//append the rest parts
if ( i > 3 ) return 0;
buffer[i] = '\0';
ipval |= atoi(buffer);
return ipval;
}
/**
* long to string ip
*
* @param ip
* @param buffer
* @return uint_t(1 for success and 0 for failed)
*/
IP2R_API uint_t long2ip(uint_t ip, char *buffer)
{
return 0;
}

138
binding/c_mmap/ip2region.h Normal file
View File

@ -0,0 +1,138 @@
/**
* ip2region header file
*
* @author chenxin<chenxin619315@gmail.com>
* @author Leo Ma<http://git.oschina.net/begeekmyfriend>
*/
#ifndef _IP2REGION_H
#define _IP2REGION_H
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
//yat, just take it as this way, 99 percent and they works find
#if ( defined(_WIN32) || defined(_WINDOWS_) || defined(__WINDOWS_) )
# define OS_WINDOW
#else
# define OS_LINUX
#endif
#ifdef OS_WINDOW
# define IP2R_API extern __declspec(dllexport)
#else
/*platform shared library statement :: unix*/
# define IP2R_API extern
#endif
#define print(str) printf("%s", str)
#define println(str) printf("%s\n", str)
/*
* memory allocation macro definition.
* cause we should use emalloc,ecalloc .ege. in php.
* so you could make it better apdat the php environment.
*/
#define IP2R_CALLOC(_bytes, _blocks) calloc(_bytes, _blocks)
#define IP2R_MALLOC(_bytes) malloc(_bytes)
#define IP2R_FREE(_ptr) free(_ptr)
typedef unsigned short ushort_t;
typedef unsigned char uchar_t;
typedef unsigned int uint_t;
typedef unsigned long ulong_t;
#define INDEX_BLOCK_LENGTH 12
#define TOTAL_HEADER_LENGTH 4096
/*
* ip2region properties struct
*/
typedef struct {
uint_t *HeaderSip; //header start ip blocks
uint_t *HeaderPtr; //header ptr blocks
uint_t headerLen; //header block number
char *dbFile; //path of db file
char *dbBinStr; //db binary string for memory search mode
uint_t dbSize;
uint_t firstIndexPtr; //first index ptr
uint_t lastIndexPtr; //last index ptr
uint_t totalBlocks; //total index blocks number
} ip2region_entry;
typedef ip2region_entry * ip2region_t;
/*
* data block
*/
typedef struct {
uint_t city_id;
char region[256];
} datablock_entry;
typedef datablock_entry * datablock_t;
/**
* create a new ip2region object
*
* @param ip2rObj
* @param dbFile path
*/
IP2R_API uint_t ip2region_create(ip2region_t, char *);
/**
* destroy the specified ip2region object
*
* @param ip2region_t
*/
IP2R_API uint_t ip2region_destroy(ip2region_t);
/**
* get the region associated with the specified ip address with memory binary search algorithm
*
* @param ip2region_t
* @param uint_t
* @param datablock_t
* @date 2016/06/30
*/
IP2R_API uint_t ip2region_binary_search(ip2region_t, uint_t, datablock_t);
IP2R_API uint_t ip2region_binary_search_string(ip2region_t, char *, datablock_t);
/**
* get the region associated with the specified ip address with b-tree algorithm
*
* @param ip2rObj
* @param ip
* @param datablock
* @return uint_t
*/
IP2R_API uint_t ip2region_btree_search(ip2region_t, uint_t, datablock_t);
IP2R_API uint_t ip2region_btree_search_string(ip2region_t, char *, datablock_t);
/**
* get a unsinged long(4bytes) from a specified buffer start from the specified offset
*
* @param buffer
* @param offset
* @return uint_t
*/
IP2R_API uint_t getUnsignedInt(char *, int);
/**
* string ip to long
*
* @param ip
* @return uint_t
*/
IP2R_API uint_t ip2long(char *);
/**
* long to string ip
*
* @param ip
* @param buffer
* @return uint_t(1 for success and 0 for failed)
*/
IP2R_API uint_t long2ip(uint_t, char *);
#endif /*end ifndef*/

View File

@ -0,0 +1,99 @@
/**
* test ip2region searcher program
*
* @author chenxin<chenxin619315@gmail.com>
* @date 2015-10-30
*/
#include "ip2region.h"
#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <sys/time.h>
#define __PRINT_ABOUT__ \
println("+-------------------------------------+"); \
println("| ip2region test program |"); \
println("| Author: chenxin619315@gmail.com. |"); \
println("| Type 'quit' to exit the program. |"); \
println("+-------------------------------------+");
//read a line from a command line.
static char *getLine( FILE *fp, char *__dst )
{
register int c;
register char *cs;
cs = __dst;
while ( ( c = getc( fp ) ) != EOF ) {
if ( c == '\n' ) break;
*cs++ = c;
}
*cs = '\0';
return ( c == EOF && cs == __dst ) ? NULL : __dst;
}
static double getTime()
{
struct timeval tv;
struct timezone tz;
gettimeofday(&tv, &tz);
return (tv.tv_sec * 1000 + ((double)tv.tv_usec)/1000);
}
int main( int argc, char **argv )
{
ip2region_entry ip2rEntry;
datablock_entry datablock;
char *dbFile = NULL, *algorithm = NULL;
char line[256];
uint_t (*func_ptr)(ip2region_t, char *, datablock_t);
double s_time, c_time;
memset(&datablock, 0x00, sizeof(datablock_entry));
if ( argc < 2 ) {
printf("Usage: a.out [ip2region db file path] [algorithm]\n");
return 0;
}
dbFile = argv[1];
algorithm = "B-tree";
func_ptr = ip2region_btree_search_string;
if ( argc >= 3 ) {
if ( strcmp(argv[2], "binary") == 0 ) {
algorithm = "File";
func_ptr = ip2region_binary_search_string;
}
}
//create a new ip2rObj
printf("+--initializing %s ... \n", algorithm);
if ( ip2region_create(&ip2rEntry, dbFile) == 0 ) {
println("Error: Fail to create the ip2region object\n");
return 0;
}
__PRINT_ABOUT__;
while ( 1 ) {
print("ip2region>> ");
getLine( stdin, line );
if ( strlen(line) < 2 ) continue;
if ( strcasecmp( line, "quit" ) == 0 ) {
println("+--Bye!");
break;
}
s_time = getTime();
func_ptr(&ip2rEntry, line, &datablock);
c_time = getTime() - s_time;
printf("%d|%s in %.5f millseconds\n", datablock.city_id, datablock.region, c_time);
}
//destory the ip2rObj
ip2region_destroy(&ip2rEntry);
return 0;
}

1
binding/nodejs/.gitignore vendored Normal file
View File

@ -0,0 +1 @@
node_modules/

View File

@ -1,3 +1,3 @@
# nodejs 客户端
现已实现同步查询,异步查询未实现,用法参考 test.js
现已实现同步查询,异步查询未实现,用法参考 ip2region.spec.js

View File

@ -0,0 +1,29 @@
// Jest Snapshot v1, https://goo.gl/fbAQLP
exports[`binarySearch 1`] = `
Object {
"city": 2163,
"region": "中国|华南|广东省|深圳市|阿里云",
}
`;
exports[`binarySearch 2`] = `
Object {
"city": 0,
"region": "未分配或者内网IP|0|0|0|0",
}
`;
exports[`should query 1`] = `
Object {
"city": 2163,
"region": "中国|华南|广东省|深圳市|阿里云",
}
`;
exports[`should query 2`] = `
Object {
"city": 0,
"region": "未分配或者内网IP|0|0|0|0",
}
`;

View File

@ -0,0 +1,15 @@
'use strict';
const ip = require('./ip2region');
const query = ip.create('../../data/ip2region.db');
it('should query', () => {
expect(query.btreeSearchSync('120.24.78.68')).toMatchSnapshot();
expect(query.btreeSearchSync('10.10.10.10')).toMatchSnapshot();
});
it('binarySearch', () => {
expect(query.binarySearchSync('120.24.78.68')).toMatchSnapshot();
expect(query.binarySearchSync('10.10.10.10')).toMatchSnapshot();
});

View File

@ -4,7 +4,7 @@
"description": "ip database ",
"main": "ip2region.js",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
"test": "jest"
},
"repository": {
"type": "git",
@ -20,5 +20,8 @@
"bugs": {
"url": "https://github.com/lionsoul2016/ip2region/issues"
},
"homepage": "https://github.com/lionsoul2016/ip2region"
"homepage": "https://github.com/lionsoul2016/ip2region",
"devDependencies": {
"jest": "^19.0.2"
}
}

View File

@ -1,15 +0,0 @@
/**
* test program
* */
var Ip2region = require('./ip2region');
// 1. create a ip2region object
var ipObj = Ip2region.create('../../data/ip2region.db');
// 2. use
//console.log(ipObj.binarySearchSync("120.24.78.68"));
console.log(ipObj.btreeSearchSync("120.24.78.68"));
// 3. destroy
Ip2region.destroy(ipObj);

1979
binding/nodejs/yarn.lock Normal file

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

Binary file not shown.

BIN
dbMaker-1.2.2.jar Normal file

Binary file not shown.