Compare commits
No commits in common. "master" and "fr_xdb_ipv6" have entirely different histories.
master
...
fr_xdb_ipv
|
|
@ -0,0 +1,2 @@
|
||||||
|
data/ipv6_source.txt filter=lfs diff=lfs merge=lfs -text
|
||||||
|
data/ip2region_v6.xdb filter=lfs diff=lfs merge=lfs -text
|
||||||
|
|
@ -6,6 +6,7 @@
|
||||||
*.log
|
*.log
|
||||||
*.la
|
*.la
|
||||||
*.so
|
*.so
|
||||||
|
*.xdb
|
||||||
*.iml
|
*.iml
|
||||||
META-INF/
|
META-INF/
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
|
@ -26,20 +27,20 @@ META-INF/
|
||||||
# vim swp file #
|
# vim swp file #
|
||||||
*.swp
|
*.swp
|
||||||
.idea
|
.idea
|
||||||
.vscode
|
|
||||||
|
|
||||||
# binding
|
# binding
|
||||||
|
/v1.0//binding/java/classes/
|
||||||
|
/v1.0//binding/java/doc/
|
||||||
|
/v1.0//binding/java/target/
|
||||||
|
/v1.0//binding/java/*.jar
|
||||||
/binding/java/classes/
|
/binding/java/classes/
|
||||||
/binding/java/doc/
|
/binding/java/doc/
|
||||||
/binding/java/target/
|
/binding/java/target/
|
||||||
/binding/java/*.jar
|
/binding/java/*.jar
|
||||||
|
|
||||||
# python
|
|
||||||
/**/__pycache__
|
|
||||||
|
|
||||||
# clang
|
# clang
|
||||||
/binding/c/xdb_searcher
|
/binding/c/xdb_searcher
|
||||||
/binding/c/test_util
|
/binding/c/util_test
|
||||||
/binding/c/cmake-build-debug
|
/binding/c/cmake-build-debug
|
||||||
|
|
||||||
# lua/luc_c
|
# lua/luc_c
|
||||||
|
|
@ -57,25 +58,24 @@ target
|
||||||
|
|
||||||
# VS ignore cases
|
# VS ignore cases
|
||||||
/**/*.sln
|
/**/*.sln
|
||||||
|
/v1.0/binding/c#/**/.vs/
|
||||||
|
/v1.0/binding/c#/**/packages
|
||||||
|
/v1.0/binding/c#/**/bin
|
||||||
|
/v1.0/binding/c#/**/obj
|
||||||
/binding/c#/**/.vs/
|
/binding/c#/**/.vs/
|
||||||
/binding/c#/**/packages
|
/binding/c#/**/packages
|
||||||
/binding/c#/**/bin
|
/binding/c#/**/bin
|
||||||
/binding/c#/**/obj
|
/binding/c#/**/obj
|
||||||
|
|
||||||
# Nodejs
|
# Nodejs
|
||||||
|
/v1.0/binding/nodejs/tests/unitTests/__snapshots__
|
||||||
|
/v1.0/binding/nodejs/coverage
|
||||||
/binding/nodejs/tests/unitTests/__snapshots__
|
/binding/nodejs/tests/unitTests/__snapshots__
|
||||||
/binding/nodejs/coverage
|
/binding/nodejs/coverage
|
||||||
/binding/nodejs/node_modules
|
/binding/nodejs/node_modules
|
||||||
/binding/nodejs/.nyc_output
|
/binding/nodejs/.nyc_output
|
||||||
/binging/nodejs/package-lock.json
|
/binging/nodejs/package-lock.json
|
||||||
|
|
||||||
# Javascript
|
|
||||||
/binding/javascript/tests/unitTests/__snapshots__
|
|
||||||
/binding/javascript/coverage
|
|
||||||
/binding/javascript/node_modules
|
|
||||||
/binding/javascript/.nyc_output
|
|
||||||
/binding/javascript/package-lock.json
|
|
||||||
|
|
||||||
# maker
|
# maker
|
||||||
## golang
|
## golang
|
||||||
/maker/golang/dbmaker
|
/maker/golang/dbmaker
|
||||||
|
|
@ -89,6 +89,3 @@ target
|
||||||
#vscode
|
#vscode
|
||||||
.vscode
|
.vscode
|
||||||
build
|
build
|
||||||
|
|
||||||
# git worktrees
|
|
||||||
.worktrees/
|
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,3 @@
|
||||||
ip2region is dual-licensed under the Apache License 2.0 and the MIT License.
|
|
||||||
|
|
||||||
You may use, distribute, and modify this software under either license,
|
|
||||||
at your option. The full text of both licenses appears below.
|
|
||||||
|
|
||||||
SPDX-License-Identifier: Apache-2.0 OR MIT
|
|
||||||
|
|
||||||
==============================================================================
|
|
||||||
Apache License
|
Apache License
|
||||||
Version 2.0, January 2004
|
Version 2.0, January 2004
|
||||||
http://www.apache.org/licenses/
|
http://www.apache.org/licenses/
|
||||||
|
|
|
||||||
125
README.md
125
README.md
|
|
@ -1,125 +0,0 @@
|
||||||
:globe_with_meridians: [中文简体](README_zh.md) | [English](README.md)
|
|
||||||
|
|
||||||
# ip2region
|
|
||||||
|
|
||||||
[ip2region](https://ip2region.net) - is an offline IP address localization library and IP localization data management framework. It supports both `IPv4` and `IPv6` with query efficiency at the 10-microsecond level. It provides `xdb` data generation and query client implementations for many mainstream programming languages.
|
|
||||||
|
|
||||||
|
|
||||||
# Features
|
|
||||||
|
|
||||||
### 1. Offline Localization Library
|
|
||||||
|
|
||||||
The project itself provides raw data for both IPv4 (`data/ipv4_source.txt`) and IPv6 (`data/ipv6_source.txt`), along with corresponding xdb files (`data/ip2region_v4.xdb` and `data/ip2region_v6.xdb`) to achieve city-level query localization. The field format is: `Country|Province|City|ISP|iso-alpha2-code`. Localization information for China is entirely in Chinese, while regional information for non-China areas is entirely in English.
|
|
||||||
|
|
||||||
### 2. Data Management Framework
|
|
||||||
|
|
||||||
`xdb` supports hundreds of millions of IP data segment rows. Region information supports full customization. The region information of the built-in data is fixed in the format: `Country|Province|City|ISP|iso-alpha2-Code`. You can append data for specific business needs to the region, such as: GPS information/International Standard Regional Codes/Zip codes, etc. In other words, you can fully use ip2region to manage your own IP localization data.
|
|
||||||
|
|
||||||
### 3. Data Deduplication and Compression
|
|
||||||
|
|
||||||
The `xdb` format generation program automatically processes the input raw data, checks and completes the merging of adjacent IP segments, and performs deduplication and compression of identical regional information.
|
|
||||||
|
|
||||||
### 4. High-Speed Query Response
|
|
||||||
|
|
||||||
Even for queries based entirely on the `xdb` file, the single query response time is at the 10-microsecond level. Memory-accelerated queries can be enabled through the following two methods:
|
|
||||||
|
|
||||||
1. `vIndex` Index Caching: Uses a fixed `512KiB` of memory to cache vector index data, reducing one disk IO operation and maintaining average query efficiency within 100 microseconds.
|
|
||||||
2. Entire `xdb` File Caching: Loads the entire `xdb` file into memory. Memory usage is equal to the `xdb` file size. There is no disk IO operation, maintaining 10-microsecond level query efficiency.
|
|
||||||
|
|
||||||
### 5. Unified Query Interface
|
|
||||||
|
|
||||||
`xdb` provides version-compatible query implementations. A unified API can simultaneously provide queries for both IPv4 and IPv6 data and return unified data.
|
|
||||||
|
|
||||||
|
|
||||||
# `xdb` Query
|
|
||||||
|
|
||||||
For API introductions, usage documentation, and test programs, please refer to the README introduction under the corresponding `searcher` query client. All query binding implementations are as follows:
|
|
||||||
|
|
||||||
| Language | Description | IPv4 Support | IPv6 Support |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| [Golang](binding/golang/README.md) | golang query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [PHP](binding/php/README.md) | php query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Java](binding/java/README.md) | java query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [C](binding/c/README.md) | C[std=c99] query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Lua_c](binding/lua_c/README.md) | lua c extension query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Lua](binding/lua/README.md) | lua query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Rust](binding/rust/README.md) | rust query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Python](binding/python/README.md) | python query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Javascript](binding/javascript/README.md) | javascript query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Csharp](binding/csharp) | csharp query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Erlang](binding/erlang/README.md) | erlang query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Nginx](binding/nginx) | nginx extension query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [C++](binding/cpp/README.md) | C++ query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Cangjie](binding/cangjie/README.md) | Cangjie query client | :white_check_mark: | :white_check_mark: |
|
|
||||||
|
|
||||||
The following toolchain implementations are contributed by community developers via third-party repositories:
|
|
||||||
|
|
||||||
| Language | Description |
|
|
||||||
| --- | --- |
|
|
||||||
| [ip2region-composer](https://github.com/zoujingli/ip2region) | php composer management client |
|
|
||||||
| [ip2region-ts](https://github.com/Steven-Qiang/ts-ip2region2) | node.js addon management client |
|
|
||||||
| [ruby-ip2region](https://github.com/jicheng1014/ruby-ip2region) | ruby xdb query client implementation |
|
|
||||||
| [Ip2regionTool](https://github.com/orestonce/Ip2regionTool) | ip2region data conversion tool |
|
|
||||||
|
|
||||||
|
|
||||||
# `xdb` Generation
|
|
||||||
|
|
||||||
For API introductions, usage documentation, and test programs, please refer to the README documents under the following `maker` generation programs:
|
|
||||||
|
|
||||||
| Language | Description | IPv4 Support | IPv6 Support |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| [Golang](maker/golang/README.md) | golang xdb generation program | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Java](maker/java/README.md) | java xdb generation program | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Python](maker/python/README.md) | python xdb generation program | :white_check_mark: | :x: |
|
|
||||||
| [Csharp](maker/csharp/README.md) | csharp xdb generation program | :white_check_mark: | :x: |
|
|
||||||
| [Rust](maker/rust/README.md) | rust xdb generation program | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [C++](maker/cpp) | C++ xdb generation program | :white_check_mark: | :white_check_mark: |
|
|
||||||
|
|
||||||
|
|
||||||
# `xdb` Update
|
|
||||||
|
|
||||||
The core of the ip2region project lies in **researching the design and implementation of IP data storage and fast querying**. The raw data `./data/ipv4_source.txt` and `./data/ipv6_source.txt` included in the project are updated irregularly. For scenarios with high requirements for data accuracy and update frequency, it is recommended to purchase commercial offline data from the [Ip2Region Community](https://ip2region.net/products/offline) or third-party vendors. You can try to update the data yourself using the following methods:
|
|
||||||
|
|
||||||
### Manual Editing and Updating
|
|
||||||
|
|
||||||
You can modify the data yourself based on the raw IP data provided by ip2region in `./data/ipv4_source.txt` and `./data/ipv6_source.txt` using the editing tools provided by ip2region. Currently, the data sources include:
|
|
||||||
|
|
||||||
1. Data provided by the ip2region community (please refer to the official account at the bottom for community notifications)
|
|
||||||
2. Project Issues tagged with `[Data_Updates]`
|
|
||||||
3. Other custom data: e.g., data provided by customers, data obtained through GPS and WIFI positioning, or legal and compliant data from other platforms.
|
|
||||||
|
|
||||||
For instructions on using the raw IP data editing tools, please refer to the README documents under the following `maker` generation programs:
|
|
||||||
|
|
||||||
| Language | Description | IPv4 Support | IPv6 Support |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| [Golang](maker/golang/README.md#xdb-data-editing) | golang IP raw data editor | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [C++](maker/cpp/README.md) | C++ IP raw data editor | :white_check_mark: | :white_check_mark: |
|
|
||||||
|
|
||||||
### Detection Automatic Update
|
|
||||||
|
|
||||||
If you want to update data via your own API or data source, you can refer to the update algorithm based on the "Detection Algorithm" shared in the following videos to write your own update program:
|
|
||||||
|
|
||||||
1. [Data Update Implementation Video Sharing - part1](https://www.bilibili.com/video/BV1934y1E7Q5/)
|
|
||||||
2. [Data Update Implementation Video Sharing - part2](https://www.bilibili.com/video/BV1pF411j7Aw/)
|
|
||||||
|
|
||||||
|
|
||||||
# Official Community
|
|
||||||
|
|
||||||
The Ip2Region official community was officially launched on `2025/06/12`. On one hand, it provides stable [commercial offline data](https://ip2region.net/products/offline) services. On the other hand, it facilitates the strengthening of the IP toolchain and data services outside the core code, such as [usage documentation](https://ip2region.net/doc/), [query testing](https://ip2region.net/search/demo), and data correction. For more information and services regarding the community, please visit the [Ip2Region Official Community](https://ip2region.net/).
|
|
||||||
|
|
||||||
|
|
||||||
# Related Remarks
|
|
||||||
|
|
||||||
### 1. xdb Technical Documents:
|
|
||||||
|
|
||||||
1. xdb Data Structure Analysis: ["ip2region xdb - Data Structure Description"](https://ip2region.net/doc/xdb/structure)
|
|
||||||
2. xdb Query Process Analysis: ["ip2region xdb - Query Process Description"](https://ip2region.net/doc/xdb/search)
|
|
||||||
3. xdb Generation Process Analysis: ["ip2region xdb - Generation Process Description"](https://ip2region.net/doc/xdb/generate)
|
|
||||||
4. xdb File Generation Tutorial: ["ip2region xdb - File Generation Tutorial"](https://ip2region.net/doc/data/xdb_make)
|
|
||||||
5. xdb Concurrent Safety Query: ["ip2region xdb - Concurrent Safety Query"](https://ip2region.net/doc/xdb/concurrent)
|
|
||||||
6. xdb Data Update Method: ["ip2region Data Update and Use of xdb Data Editor"](https://mp.weixin.qq.com/s/cZH5qIn4E5rQFy6N32RCzA)
|
|
||||||
|
|
||||||
### 3. Technical Information Blogs
|
|
||||||
|
|
||||||
1. WeChat Official Account - lionsoul-org, the author's active technical sharing channel
|
|
||||||
2. [Ip2Region Official Community](https://ip2region.net)
|
|
||||||
117
README_zh.md
117
README_zh.md
|
|
@ -1,117 +0,0 @@
|
||||||
:globe_with_meridians: [中文简体](README_zh.md) | [English](README.md)
|
|
||||||
|
|
||||||
# ip2region
|
|
||||||
|
|
||||||
[ip2region](https://ip2region.net) - 是一个离线IP地址定位库和IP定位数据管理框架，同时支持 `IPv4` 和 `IPv6` ，10微秒级别的查询效率，提供了众多主流编程语言的 `xdb` 数据生成和查询客户端实现。
|
|
||||||
|
|
||||||
|
|
||||||
# 项目特性
|
|
||||||
|
|
||||||
### 1、离线定位库
|
|
||||||
|
|
||||||
项目本身同时了提供了一份 IPv4 (`data/ipv4_source.txt`) 和 IPv6 (`data/ipv6_source.txt`) 的原始数据和对应的 xdb 文件(`data/ip2region_v4.xdb` 和 `data/ip2region_v6.xdb`) 用于实现精确到城市的查询定位功能，字段格式为：`国家|省份|城市|ISP|iso-alpha2-code(国家两字母简称)`，中国的定位信息全部为中文，非中国地区的地域信息全部为英文。
|
|
||||||
|
|
||||||
### 2、数据管理框架
|
|
||||||
|
|
||||||
`xdb` 支持亿级别的 IP 数据段行数，region 信息支持完全自定义，自带数据的 region 信息固定了格式为：`国家|省份|城市|ISP|iso-alpha2-Code`，你可以在 region 中追加特定业务需求的数据，例如：GPS信息/国际统一地域信息编码/邮编等。也就是你完全可以使用 ip2region 来管理你自己的 IP 定位数据。
|
|
||||||
|
|
||||||
### 3、数据去重和压缩
|
|
||||||
|
|
||||||
`xdb` 格式生成程序会自动处理输入的原始数据，检查并且完成相连 IP 段的的合并以及相同地域信息的去重和压缩。
|
|
||||||
|
|
||||||
### 4、极速查询响应
|
|
||||||
|
|
||||||
即使是完全基于 `xdb` 文件的查询，单次查询响应时间在十微秒级别，可通过如下两种方式开启内存加速查询：
|
|
||||||
|
|
||||||
1. `vIndex` 索引缓存 ：使用固定的 `512KiB` 的内存空间缓存 vector index 数据，减少一次 IO 磁盘操作，保持平均查询效率稳定在100微秒之内。
|
|
||||||
2. `xdb` 整个文件缓存：将整个 `xdb` 文件全部加载到内存，内存占用等同于 `xdb` 文件大小，无磁盘 IO 操作，保持10微秒级别的查询效率。
|
|
||||||
|
|
||||||
### 5、统一的查询接口
|
|
||||||
|
|
||||||
`xdb` 提供了版本兼容的查询实现，一个统一的 API 可以同时提供对 IPv4 和 IPv6 数据的查询并且返回统一的数据。
|
|
||||||
|
|
||||||
|
|
||||||
# `xdb` 查询
|
|
||||||
|
|
||||||
API 介绍，使用文档和测试程序请参考对应 `searcher` 查询客户端下的 README 介绍，全部查询 binding 实现情况如下：
|
|
||||||
| 编程语言 | 描述 | IPv4 支持 | IPv6 支持 |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| [Golang](binding/golang/README_zh.md) | golang 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [PHP](binding/php/README_zh.md) | php 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Java](binding/java/README_zh.md) | java 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [C](binding/c/README_zh.md) | C[std=c99] 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Lua_c](binding/lua_c/README_zh.md) | lua c 扩展查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Lua](binding/lua/README_zh.md) | lua 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Rust](binding/rust/README_zh.md) | rust 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Python](binding/python/README_zh.md) | python 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Javascript](binding/javascript/README_zh.md) | javascript 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Csharp](binding/csharp) | csharp 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Erlang](binding/erlang/README_zh.md) | erlang 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Nginx](binding/nginx) | nginx 扩展查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [C++](binding/cpp/README_zh.md) | C++ 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Cangjie](binding/cangjie/README.md) | 仓颉 查询客户端 | :white_check_mark: | :white_check_mark: |
|
|
||||||
|
|
||||||
|
|
||||||
以下工具链实现由社区开发者通过第三方仓库贡献：
|
|
||||||
| 编程语言 | 描述 |
|
|
||||||
| --- | --- |
|
|
||||||
| [ip2region-composer](https://github.com/zoujingli/ip2region) | php composer 管理客户端 |
|
|
||||||
| [ip2region-ts](https://github.com/Steven-Qiang/ts-ip2region2) | node.js addon 管理客户端|
|
|
||||||
| [ruby-ip2region](https://github.com/jicheng1014/ruby-ip2region) | ruby xdb 查询客户端实现 |
|
|
||||||
| [Ip2regionTool](https://github.com/orestonce/Ip2regionTool) | ip2region 数据转换工具 |
|
|
||||||
|
|
||||||
|
|
||||||
# `xdb` 生成
|
|
||||||
|
|
||||||
API 介绍，使用文档和测试程序请参考如下 `maker` 生成程序下的 README 文档：
|
|
||||||
|
|
||||||
| 编程语言 | 描述 | IPv4 支持 | IPv6 支持 |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| [Golang](maker/golang/README_zh.md) | golang xdb 生成程序 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Java](maker/java/README_zh.md) | java xdb 生成程序 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [Python](maker/python/README_zh.md) | python xdb 生成程序 | :white_check_mark: | :x: |
|
|
||||||
| [Csharp](maker/csharp/README_zh.md) | csharp xdb 生成程序 | :white_check_mark: | :x: |
|
|
||||||
| [Rust](maker/rust/README_zh.md) | rust xdb 生成程序 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [C++](maker/cpp) | C++ xdb 生成程序 | :white_check_mark: | :white_check_mark: |
|
|
||||||
|
|
||||||
|
|
||||||
# `xdb` 更新
|
|
||||||
|
|
||||||
ip2region 项目的核心在于 <b>研究 IP 数据的存储和快速查询的设计和实现</b>， 项目自带的 `./data/ipv4_source.txt` 和 `./data/ipv6_source.txt` 原始数据不定期更新，对于数据精度和更新频率要求很高的使用场景建议到 [Ip2Region社区](https://ip2region.net/products/offline) 或者第三方购买商用离线数据，你可以使用如下几种方式来尝试自己更新数据：
|
|
||||||
|
|
||||||
### 手动编辑更新
|
|
||||||
你可以基于 ip2region 自带的 `./data/ipv4_source.txt` 和 `./data/ipv6_source.txt` 原始 IP 数据用 ip2region 提供的编辑工具来自己修改，目前数据源有如下几种方式：
|
|
||||||
1. ip2region 社区提供的数据（请参考地底部的公众号关注社区通知）
|
|
||||||
2. ip2region Github/Gitee 中带有 `[数据源补充]` 标签的 Issue
|
|
||||||
3. 其他自定义数据：例如客户提供的数据，或者通过 GPS 和 WIFI 定位得到的数据，或者来自其他平台的合法合规的数据
|
|
||||||
|
|
||||||
原始 IP 数据编辑工具使用方法请参考如下的 `maker` 生成程序下的 README 文档：
|
|
||||||
| 编程语言 | 描述 | IPv4 支持 | IPv6 支持 |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| [Golang](maker/golang/README_zh.md#xdb-数据编辑) | golang IP 原始数据编辑器 | :white_check_mark: | :white_check_mark: |
|
|
||||||
| [C++](maker/cpp) | C++ IP 原始数据编辑器 | :white_check_mark: | :white_check_mark: |
|
|
||||||
|
|
||||||
|
|
||||||
### 检测自动更新
|
|
||||||
如果你想通过你自己的 API 或数据源来更新数据，你可以参考以下视频分享的 `基于检测算法` 的更新算法来自己编写一个更新程序：
|
|
||||||
1. [数据更新实现视频分享 - part1](https://www.bilibili.com/video/BV1934y1E7Q5/)
|
|
||||||
2. [数据更新实现视频分享 - part2](https://www.bilibili.com/video/BV1pF411j7Aw/)
|
|
||||||
|
|
||||||
|
|
||||||
# 官方社区
|
|
||||||
Ip2Region 官方社区正式上线于 `2025/06/12` 日，一方面提供了稳定的 [商用离线数据](https://ip2region.net/products/offline) 服务，另一方面便于在核心代码外强化 IP 工具链和数据服务，例如 [使用文档](https://ip2region.net/doc/)，[查询测试](https://ip2region.net/search/demo)，数据纠错等，更多关于社区的信息和服务请访问 [Ip2Region 官方社区](https://ip2region.net/)。
|
|
||||||
|
|
||||||
|
|
||||||
# 相关备注
|
|
||||||
|
|
||||||
### 1、xdb 技术文档：
|
|
||||||
1. xdb 数据结构分析：[“ip2region xdb-数据结构描述“](https://ip2region.net/doc/xdb/structure)
|
|
||||||
2. xdb 查询过程分析：[“ip2region xdb-查询过程描述”](https://ip2region.net/doc/xdb/search)
|
|
||||||
3. xdb 生成过程分析：[“ip2region xdb-生成过程描述”](https://ip2region.net/doc/xdb/generate)
|
|
||||||
4. xdb 文件生成教程：[“ip2region xdb-文件生成教程”](https://ip2region.net/doc/data/xdb_make)
|
|
||||||
5. xdb 并发安全查询：[“ip2region xdb-并发安全查询”](https://ip2region.net/doc/xdb/concurrent)
|
|
||||||
6. xdb 数据更新方法：[“ip2region 数据更新和 xdb 数据编辑器的使用”](https://mp.weixin.qq.com/s/cZH5qIn4E5rQFy6N32RCzA)
|
|
||||||
|
|
||||||
### 3、技术信息博客
|
|
||||||
1. 微信公众号 - lionsoul-org，作者活跃的技术分享渠道
|
|
||||||
2. [Ip2Region 官方社区](https://ip2region.net)
|
|
||||||
|
|
@ -0,0 +1,115 @@
|
||||||
|
# Ip2region 是什么
|
||||||
|
|
||||||
|
[ip2region](https://ip2region.net) - 是一个离线IP地址定位库和IP定位数据管理框架，同时支持`IPv4`和`IPv6`，10微秒级别的查询效率，提供了众多主流编程语言的 `xdb` 数据生成和查询客户端实现。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# Ip2region 特性
|
||||||
|
|
||||||
|
### 1、离线定位库
|
||||||
|
|
||||||
|
项目本身同时了提供了一份 IPv4 和 IPv6 的原始数据和对应的 xdb 文件(`ip2region_v4.xdb` 和 `ip2region_v6.xdb`) 用于实现精确到城市的的查询定位功能。
|
||||||
|
|
||||||
|
### 2、数据管理框架
|
||||||
|
|
||||||
|
`xdb` 支持亿级别的 IP 数据段行数，默认的 region 信息都固定了格式：`国家|省份|城市|ISP`，缺省的地域信息默认是0。
|
||||||
|
region 信息支持完全自定义，例如：你可以在 region 中追加特定业务需求的数据，例如：GPS信息/国际统一地域信息编码/邮编等。也就是你完全可以使用 ip2region 来管理你自己的 IP 定位数据。
|
||||||
|
|
||||||
|
### 3、数据去重和压缩
|
||||||
|
|
||||||
|
`xdb` 格式生成程序会自动处理输入的原始数据，检查并且完成相连 IP 段的的合并以及相同地域信息的去重和压缩。
|
||||||
|
|
||||||
|
### 4、极速查询响应
|
||||||
|
|
||||||
|
即使是完全基于 `xdb` 文件的查询，单次查询响应时间在十微秒级别，可通过如下两种方式开启内存加速查询：
|
||||||
|
|
||||||
|
1. `vIndex` 索引缓存 ：使用固定的 `512KiB` 的内存空间缓存 vector index 数据，减少一次 IO 磁盘操作，保持平均查询效率稳定在100微秒之内。
|
||||||
|
2. `xdb` 整个文件缓存：将整个 `xdb` 文件全部加载到内存，内存占用等同于 `xdb` 文件大小，无磁盘 IO 操作，保持10微秒级别的查询效率。
|
||||||
|
|
||||||
|
### 5、统一的查询接口
|
||||||
|
|
||||||
|
`xdb` 提供了版本兼容的查询实现，一个统一的 API 可以同时提供对 IPv4 和 IPv6 数据的查询并且返回统一的数据。
|
||||||
|
|
||||||
|
# `xdb` 数据查询
|
||||||
|
|
||||||
|
API 介绍，使用文档和测试程序请参考对应 `searcher` 查询客户端下的 ReadMe 介绍，全部查询 binding 实现情况如下：
|
||||||
|
| 编程语言 | 描述 | IPv4 支持 | IPv6 支持 | 贡献者 |
|
||||||
|
|:---------------------------------|:----------------------------|:-------------------|:------------------ |:--------------------------------------------------|
|
||||||
|
| [Golang](binding/golang) | golang xdb 查询客户端 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
|
||||||
|
| [PHP](binding/php) | php xdb 查询客户端 | :white_check_mark: | :soon: | [Lion](https://github.com/lionsoul2014) |
|
||||||
|
| [Java](binding/java) | java xdb 查询客户端 | :white_check_mark: | :soon: | [Lion](https://github.com/lionsoul2014) |
|
||||||
|
| [Lua](binding/lua) | lua xdb 查询客户端 | :white_check_mark: | :soon: | [Lion](https://github.com/lionsoul2014) |
|
||||||
|
| [ANSI C](binding/c) | ANSC c xdb 查询客户端 | :white_check_mark: | :soon: | [Lion](https://github.com/lionsoul2014) |
|
||||||
|
| [Lua_c](binding/lua_c) | lua c 扩展 xdb 查询客户端 | :white_check_mark: | :soon: | [Lion](https://github.com/lionsoul2014) |
|
||||||
|
| [Rust](binding/rust) | rust xdb 查询客户端 | :white_check_mark: | :x: | [gongzhengyang](https://github.com/gongzhengyang) |
|
||||||
|
| [Python](binding/python) | python xdb 查询客户端 | :white_check_mark: | :x: | [厉害的花花](https://github.com/luckydog6132) |
|
||||||
|
| [Nodejs](binding/nodejs) | nodejs xdb 查询客户端 | :white_check_mark: | :x: | [Wu Jian Ping](https://github.com/wujjpp) |
|
||||||
|
| [Csharp](binding/csharp) | csharp xdb 查询客户端 | :white_check_mark: | :x: | [Alen Lee](https://github.com/malus2077) |
|
||||||
|
| [Erlang](binding/erlang) | erlang xdb 查询客户端 | :white_check_mark: | :x: | [malou](https://github.com/malou996) |
|
||||||
|
| [Nginx](binding/nginx) | nginx 扩展 xdb 查询客户端 | :white_check_mark: | :x: | [Wu Jian Ping](https://github.com/wujjpp) |
|
||||||
|
| [C++](binding/cpp) | C++ xdb 查询客户端 | :white_check_mark: | :x: | [Yunbin Liu](https://github.com/liuyunbin) |
|
||||||
|
| [Typescript](binding/typescript) | Typescript xdb 查询客户端 | :white_check_mark: | :x: | [Alan Lee](https://github.com/malus2077) |
|
||||||
|
|
||||||
|
|
||||||
|
以下工具链实现由社区开发者通过第三方仓库贡献：
|
||||||
|
| 编程语言 | 描述 | 贡献者 |
|
||||||
|
|:--------------------------------------------------------------- |:------------------------|:-----------------------------------------------|
|
||||||
|
| [ruby-ip2region](https://github.com/jicheng1014/ruby-ip2region) | ruby xdb 查询客户端实现 | [jicheng1014](https://github.com/jicheng1014) |
|
||||||
|
| [Ip2regionTool](https://github.com/orestonce/Ip2regionTool) | ip2region 数据转换工具 | [orestonce](https://github.com/orestonce) |
|
||||||
|
|
||||||
|
|
||||||
|
# `xdb` 数据生成
|
||||||
|
|
||||||
|
API 介绍，使用文档和测试程序请参考如下 `maker` 生成程序下的 ReadMe 文档：
|
||||||
|
|
||||||
|
| 编程语言 | 描述 | IPv4 支持 | IPv6 支持 | 贡献者 |
|
||||||
|
| :---------------------- | :------------------- |:-------------------|:-------------------| :----------------------------------------- |
|
||||||
|
| [Golang](maker/golang) | golang xdb 生成程序 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
|
||||||
|
| [Java](maker/java) | java xdb 生成程序 | :white_check_mark: | :soon: | [Lion](https://github.com/lionsoul2014) |
|
||||||
|
| [Python](maker/python) | python xdb 生成程序 | :white_check_mark: | :x: | [leolin49](https://github.com/leolin49) |
|
||||||
|
| [Csharp](maker/csharp) | csharp xdb 生成程序 | :white_check_mark: | :x: | [Alan Lee](https://github.com/malus2077) |
|
||||||
|
| [Rust](maker/rust) | rust xdb 生成程序 | :white_check_mark: | :x: | [KevinWang](https://github.com/KevinWL) |
|
||||||
|
| [C++](maker/cpp) | C++ xdb 生成程序 | :white_check_mark: | :x: | [Yunbin Liu](https://github.com/liuyunbin) |
|
||||||
|
|
||||||
|
|
||||||
|
# `xdb` 数据更新
|
||||||
|
|
||||||
|
ip2region 项目的核心在于 <b>研究 IP 数据的存储和快速查询的设计和实现</b>， 项目自带的 `./data/ipv4_source.txt` 和 `./data/ipv6_source.txt` 原始数据不会再提供更新，对于数据精度和更新频率要求很高的使用场景建议到 [Ip2Region社区](https://ip2region.net/products/offline) 或者第三方购买商用离线数据，你可以使用如下几种方式来尝试自己更新数据：
|
||||||
|
|
||||||
|
### 手动编辑更新
|
||||||
|
你可以基于 ip2region 自带的 `./data/ipv4_source.txt` 和 `./data/ipv6_source.txt` 原始 IP 数据用 ip2region 提供的编辑工具来自己修改，目前数据源有如下几种方式：
|
||||||
|
1. ip2region 社区提供的数据（请参考地底部的公众号关注社区通知）
|
||||||
|
2. ip2region Github/Gitee 中带有 `[数据源补充]` 标签的 Issue
|
||||||
|
3. 其他自定义数据：例如客户提供的数据，或者通过 GPS 和 WIFI 定位得到的数据，或者来自其他平台的合法合规的数据
|
||||||
|
|
||||||
|
原始 IP 数据编辑工具使用方法请参考如下的 `maker` 生成程序下的 ReadMe 文档：
|
||||||
|
| 编程语言 | 描述 | IPv4 支持 | IPv6 支持 | 贡献者 |
|
||||||
|
|:------------------------------------|:-------------------------|:-------------------|:-------------------|:-------------------------------------------|
|
||||||
|
| [Golang](maker/golang#xdb-数据编辑) | golang IP 原始数据编辑器 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
|
||||||
|
| [Java](maker/java#xdb-数据编辑) | java IP 原始数据编辑器 | :white_check_mark: | :soon: | [Lion](https://github.com/lionsoul2014) |
|
||||||
|
| [C++](maker/cpp#xdb-数据编辑) | C++ IP 原始数据编辑器 | :white_check_mark: | :x: | [Yunbin Liu](https://github.com/liuyunbin) |
|
||||||
|
|
||||||
|
|
||||||
|
### 检测自动更新
|
||||||
|
如果你想通过你自己的 API 或数据源来更新数据，你可以参考以下视频分享的 `基于检测算法` 的更新算法来自己编写一个更新程序：
|
||||||
|
1. [数据更新实现视频分享 - part1](https://www.bilibili.com/video/BV1934y1E7Q5/)
|
||||||
|
2. [数据更新实现视频分享 - part2](https://www.bilibili.com/video/BV1pF411j7Aw/)
|
||||||
|
|
||||||
|
# 官方社区
|
||||||
|
Ip2Region 官方社区正式上线于 `2025/06/12` 日，一方面提供了稳定的 [商用离线数据](https://ip2region.net/products/offline) 服务，另一方面便于在核心代码外强化 IP 工具链和数据服务，例如 [使用文档](https://ip2region.net/doc/)，[查询测试](https://ip2region.net/search/demo)，数据纠错等，更多关于社区的信息和服务请访问 [Ip2Region 官方社区](https://ip2region.net/)。
|
||||||
|
|
||||||
|
# 相关备注
|
||||||
|
|
||||||
|
### 1、并发查询必读
|
||||||
|
xdb 整个缓存的查询都 <b>是</b> 并发安全的，基于文件的查询都 <b>不是</b> 并发安全的实现，不同进程/线程/协程需要通过创建不同的查询对象来安全使用，并发量很大的情况下，基于文件查询的方式可能会导致打开文件数过多的错误，请修改内核的最大允许打开文件数(fs.file-max=一个更高的值)，或者将整个xdb加载到内存进行安全并发使用。
|
||||||
|
|
||||||
|
### 2、技术资源分享
|
||||||
|
1. xdb 数据结构分析：[“ip2region xdb-数据结构描述“](https://ip2region.net/doc/xdb/ipv4_structure)
|
||||||
|
2. xdb 查询过程分析：[“ip2region xdb-查询过程描述”](https://ip2region.net/doc/xdb/ipv4_search)
|
||||||
|
3. xdb 生成过程分析：[“ip2region xdb-生成过程描述”](https://ip2region.net/doc/xdb/ipv4_generate)
|
||||||
|
4. xdb 文件生成教程：[“ip2region xdb-文件生成教程”](https://ip2region.net/doc/data/ipv4_xdb_make)
|
||||||
|
5. xdb 数据更新方法：[“ip2region 数据更新和 xdb 数据编辑器的使用”](https://mp.weixin.qq.com/s/cZH5qIn4E5rQFy6N32RCzA)
|
||||||
|
|
||||||
|
### 3、技术信息博客
|
||||||
|
1. [Ip2Region 官方社区](https://ip2region.net)
|
||||||
|
2. 请先关注微信公众号 lionsoul-org (狮子的魂)
|
||||||
|
|
@ -1,27 +1,24 @@
|
||||||
all: xdb_searcher test_util
|
all: xdb_searcher util_test
|
||||||
|
|
||||||
xdb_searcher: xdb_api.h xdb_util.c xdb_searcher.c main.c
|
xdb_searcher: xdb_searcher.h xdb_searcher.c main.c
|
||||||
gcc -std=c99 -Wall -O2 -I./ xdb_util.c xdb_searcher.c main.c -o xdb_searcher
|
gcc -O2 -I./ xdb_searcher.c main.c -o xdb_searcher
|
||||||
|
|
||||||
test_util: xdb_api.h xdb_util.c test_util.c
|
util_test: xdb_searcher.h xdb_searcher.c util_test.c
|
||||||
gcc -std=c99 -Wall -O2 -I./ xdb_util.c test_util.c -o test_util
|
gcc -O2 -I./ xdb_searcher.c util_test.c -o util_test
|
||||||
|
|
||||||
xdb_searcher.o: xdb_searcher.c
|
xdb_searcher.o: xdb_searcher.c
|
||||||
gcc -std=c99 -Wall -c xdb_searcher.c
|
gcc -c -o xdb_searcher.o xdb_searcher.c
|
||||||
|
|
||||||
xdb_util.o: xdb_util.c
|
xdb_searcher_lib: xdb_searcher.o
|
||||||
gcc -std=c99 -Wall -c xdb_util.c
|
|
||||||
|
|
||||||
xdb_searcher_lib: xdb_util.o xdb_searcher.o
|
|
||||||
mkdir -p build/lib
|
mkdir -p build/lib
|
||||||
mkdir -p build/include
|
mkdir -p build/include
|
||||||
ar -rc build/lib/libxdb.a `find . -name "*.o"`
|
ar -rc build/lib/libxdb_searcher.a `find . -name *.o`
|
||||||
cp xdb_api.h build/include
|
cp xdb_searcher.h build/include
|
||||||
|
|
||||||
clean:
|
clean:
|
||||||
find ./ -name \*.o | xargs rm -f
|
find ./ -name \*.o | xargs rm -f
|
||||||
find ./ -name test_util | xargs rm -f
|
find ./ -name util_test | xargs rm -f
|
||||||
find ./ -name xdb_searcher | xargs rm -f
|
find ./ -name xdb_searcher | xargs rm -f
|
||||||
rm -rf build
|
rm -rf build
|
||||||
|
|
||||||
.PHONY: all clean xdb_searcher test_util
|
.PHONY: all clean xdb_searcher util_test
|
||||||
|
|
|
||||||
|
|
@ -1,370 +0,0 @@
|
||||||
:globe_with_meridians: [中文简体](README_zh.md) | [English](README.md)
|
|
||||||
|
|
||||||
# ip2region c Query Client
|
|
||||||
|
|
||||||
# Usage
|
|
||||||
|
|
||||||
### About Query API
|
|
||||||
|
|
||||||
The prototype of the Query API is as follows:
|
|
||||||
|
|
||||||
```c
|
|
||||||
// Query via string IP
|
|
||||||
int xdb_search_by_string(xdb_searcher_t *, const string_ip_t *, xdb_region_buffer_t *);
|
|
||||||
// Query via binary IP returned by xdb_parse_ip
|
|
||||||
int xdb_search(xdb_searcher_t *, const bytes_ip_t *, int, xdb_region_buffer_t *);
|
|
||||||
```
|
|
||||||
|
|
||||||
If the query fails, a non-`0` error code will be returned. If the query is successful, the `region` information string can be obtained from `xdb_region_buffer_t`. If the input IP cannot be found, `xdb_region_buffer_t` will receive an empty string `""`.
|
|
||||||
|
|
||||||
### About IPv4 and IPv6
|
|
||||||
|
|
||||||
This xdb query client implementation supports both IPv4 and IPv6 queries. The usage is as follows:
|
|
||||||
|
|
||||||
```c
|
|
||||||
#include "xdb_api.h";
|
|
||||||
|
|
||||||
// For IPv4: Set xdb path to the v4 xdb file, specify IP version as IPv4
|
|
||||||
const char *db_path = "../../data/ip2region_v4.xdb"; // or your ipv4 xdb path
|
|
||||||
xdb_version_t *version = XDB_IPv4;
|
|
||||||
|
|
||||||
// For IPv6: Set xdb path to the v6 xdb file, specify IP version as IPv6
|
|
||||||
const char *db_path = "../../data/ip2region_v6.xdb"; // or your ipv6 xdb path
|
|
||||||
xdb_version_t *version = XDB_IPv6;
|
|
||||||
|
|
||||||
// The IP version of the xdb specified by db_path must be consistent with the version, otherwise an error will occur during query execution
|
|
||||||
// Note: The following demonstration directly uses db_path and version variables
|
|
||||||
```
|
|
||||||
|
|
||||||
### XDB File Verification
|
|
||||||
|
|
||||||
It is recommended that you proactively verify the applicability of the xdb file, as some future new features may cause the current Searcher version to be incompatible with the xdb file you are using. Verification can avoid unpredictable errors during runtime. You do not need to verify every time; for example, verify when the service starts or manually call a command to confirm version matching. Do not run verification every time a Searcher is created, as this will affect query response speed, especially in high-concurrency scenarios.
|
|
||||||
|
|
||||||
```c
|
|
||||||
#include "xdb_api.h";
|
|
||||||
|
|
||||||
int errcode = xdb_verify(db_path);
|
|
||||||
if ($err != 0) {
|
|
||||||
// Applicability verification failed!!!
|
|
||||||
// The current query client implementation is not suitable for querying the xdb file specified by db_path.
|
|
||||||
// You should stop the service and use a suitable xdb file or upgrade to a Searcher implementation compatible with db_path.
|
|
||||||
printf("failed to verify xdb file `%s`, errcode: %d\n", db_path, errcode);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Verification passed, the current Searcher can be safely used for query operations on the xdb pointed to by dbPath
|
|
||||||
```
|
|
||||||
|
|
||||||
### File-Based Query
|
|
||||||
|
|
||||||
```c
|
|
||||||
#include <stdio.h>
|
|
||||||
#include "xdb_api.h"
|
|
||||||
|
|
||||||
int main(int argc, char *argv[]) {
|
|
||||||
xdb_searcher_t searcher;
|
|
||||||
char region_buffer[512] = {'\0'};
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
|
|
||||||
// Initialize region_buffer_t using region_buffer from stack space
|
|
||||||
int err = xdb_region_buffer_init(®ion, region_buffer, sizeof(region_buffer));
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the region buffer with errcode=%d\n", err);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Initialize winsock when the service starts; no need to call repeatedly, only needed on Windows systems
|
|
||||||
err = xdb_init_winsock();
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the winsock with errno=%d\n", err);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 1. Initialize xdb query object from db_path.
|
|
||||||
// @Note: Use the db_path and version described above to create the searcher
|
|
||||||
err = xdb_new_with_file_only(version, &searcher, db_path);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to create xdb searcher from `%s` with errno=%d\n", db_path, err);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 2. Call search API to query, both IPv4 and IPv6 are supported.
|
|
||||||
const char *ip_string = "1.2.3.4";
|
|
||||||
// ip_string = "240e:3b7:3272:d8d0:db09:c067:8d59:539e"; // IPv6
|
|
||||||
|
|
||||||
long cost_time = 0, s_time = xdb_now();
|
|
||||||
err = xdb_search_by_string(&searcher, ip_string, ®ion);
|
|
||||||
cost_time = (int) (xdb_now() - s_time);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed search(%s) with errno=%d\n", ip_string, err);
|
|
||||||
} else {
|
|
||||||
printf("{region: %s, took: %d μs}", region.value, cost_time);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Clean up memory resources for region info; must be called after every search
|
|
||||||
xdb_region_buffer_free(®ion);
|
|
||||||
|
|
||||||
// Note: For concurrent use, each thread needs to define and initialize its own searcher query object independently.
|
|
||||||
|
|
||||||
// 3. Close xdb searcher
|
|
||||||
xdb_close(&searcher);
|
|
||||||
xdb_clean_winsock(); // Call on Windows
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Caching `VectorIndex`
|
|
||||||
|
|
||||||
We can pre-load VectorIndex data from the xdb file and cache it globally. Using the global VectorIndex cache every time a Searcher object is created can reduce a fixed IO operation, thereby accelerating queries and reducing IO pressure.
|
|
||||||
|
|
||||||
```c
|
|
||||||
#include <stdio.h>
|
|
||||||
#include "xdb_api.h"
|
|
||||||
|
|
||||||
int main(int argc, char *argv[]) {
|
|
||||||
xdb_vector_index_t *v_index;
|
|
||||||
xdb_searcher_t searcher;
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
|
|
||||||
// Initialize region_buffer with NULL to let it manage memory allocation automatically
|
|
||||||
int err = xdb_region_buffer_init(®ion, NULL, 0);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the region buffer with errcode=%d\n", err);
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Initialize winsock when the service starts; no need to call repeatedly, only needed on Windows systems
|
|
||||||
err = xdb_init_winsock();
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the winsock with errno=%d\n", err);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 1. Load VectorIndex from the db_path described above.
|
|
||||||
// Obtain v_index to create a global cache for subsequent repeated use.
|
|
||||||
// Note: v_index does not need to be loaded every time; it is recommended to load it once at service startup as a global resource.
|
|
||||||
v_index = xdb_load_vector_index_from_file(db_path);
|
|
||||||
if (v_index == NULL) {
|
|
||||||
printf("failed to load vector index from `%s`\n", db_path);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 2. Use the global VectorIndex variable to create an xdb searcher with VectorIndex cache.
|
|
||||||
// @Note: Use the db_path and version described above to create the searcher
|
|
||||||
err = xdb_new_with_vector_index(version, &searcher, db_path, v_index);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to create vector index cached searcher with errcode=%d\n", err);
|
|
||||||
return 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
// 3. Call search API to query, both IPv4 and IPv6 are supported
|
|
||||||
const char *ip_string = "1.2.3.4";
|
|
||||||
// ip_string = "240e:3b7:3272:d8d0:db09:c067:8d59:539e"; // IPv6
|
|
||||||
|
|
||||||
long cost_time = 0, s_time = xdb_now();
|
|
||||||
err = xdb_search_by_string(&searcher, ip_string, ®ion);
|
|
||||||
cost_time = (int) (xdb_now() - s_time);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed search(%s) with errno=%d\n", ip_string, err);
|
|
||||||
} else {
|
|
||||||
printf("{region: %s, took: %d μs}", region.value, cost_time);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Clean up memory resources for region info; must be called after every search
|
|
||||||
xdb_region_buffer_free(®ion);
|
|
||||||
|
|
||||||
|
|
||||||
// Note: For concurrent use, each thread needs to define and initialize its own searcher query object independently.
|
|
||||||
|
|
||||||
// 4. Close xdb searcher; if the service is being shut down, the memory for v_index also needs to be freed.
|
|
||||||
xdb_close(&searcher);
|
|
||||||
xdb_close_vector_index(v_index);
|
|
||||||
xdb_clean_winsock();
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Caching the Entire `xdb` File
|
|
||||||
|
|
||||||
We can also pre-load the entire xdb file into memory and then create a query object based on this data to achieve fully memory-based queries, similar to the previous memory search.
|
|
||||||
|
|
||||||
```c
|
|
||||||
#include <stdio.h>
|
|
||||||
#include "xdb_api.h"
|
|
||||||
|
|
||||||
int main(int argc, char *argv[]) {
|
|
||||||
xdb_content_t *c_buffer;
|
|
||||||
xdb_searcher_t searcher;
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
|
|
||||||
// Initialize region_buffer with NULL to let it manage memory allocation automatically
|
|
||||||
int err = xdb_region_buffer_init(®ion, NULL, 0);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the region buffer with errcode=%d\n", err);
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
// Initialize winsock when the service starts; no need to call repeatedly, only needed on Windows systems
|
|
||||||
err = xdb_init_winsock();
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the winsock with errno=%d\n", err);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 1. Load the entire xdb data from the db_path described above.
|
|
||||||
c_buffer = xdb_load_content_from_file(db_path);
|
|
||||||
if (v_index == NULL) {
|
|
||||||
printf("failed to load xdb content from `%s`\n", db_path);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 2. Use the global c_buffer variable to create a fully memory-based xdb query object.
|
|
||||||
// @Note: Use the version described above to create the searcher.
|
|
||||||
err = xdb_new_with_buffer(version, &searcher, c_buffer);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to create content cached searcher with errcode=%d\n", err);
|
|
||||||
return 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 3. Call search API to query, both IPv4 and IPv6 are supported
|
|
||||||
const char *ip_string = "1.2.3.4";
|
|
||||||
// ip_string = "240e:3b7:3272:d8d0:db09:c067:8d59:539e"; // IPv6
|
|
||||||
|
|
||||||
long cost_time = 0, s_time = xdb_now();
|
|
||||||
err = xdb_search_by_string(&searcher, ip_string, ®ion);
|
|
||||||
cost_time = (int) (xdb_now() - s_time);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed search(%s) with errno=%d\n", ip_string, err);
|
|
||||||
} else {
|
|
||||||
printf("{region: %s, took: %d μs}", region.value, cost_time);
|
|
||||||
}
|
|
||||||
|
|
||||||
// Clean up memory resources for region info; must be called after every search
|
|
||||||
xdb_region_buffer_free(®ion);
|
|
||||||
|
|
||||||
|
|
||||||
// Note: For concurrent use, xdb query objects created this way can be safely used for concurrency.
|
|
||||||
// It is recommended to create them when the service starts and then use them safely in parallel until the service shuts down.
|
|
||||||
|
|
||||||
// 4. Close xdb searcher; memory for c_buffer needs to be freed when shutting down the service.
|
|
||||||
xdb_close(&searcher);
|
|
||||||
xdb_close_content(c_buffer);
|
|
||||||
xdb_clean_winsock();
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### About Storage of Location Information
|
|
||||||
|
|
||||||
In older implementations, search-related functions relied on a specified `region_buffer` memory to store location information, which had significant limitations.
|
|
||||||
The new implementation provides an `xdb_region_buffer_t` object to manage these memory allocations. You can still specify a fixed `region_buffer` to create memory management for the region; this is suitable when the maximum length of your location information is known, which helps reduce memory fragmentation during runtime. If the length of the location information is uncertain or if your program is not suited for pre-allocating a block of memory, you can initialize `xdb_region_buffer_t` by specifying `NULL`. In this case, the object will automatically manage memory allocation, making it suitable for storing location information of any length, though this approach will certainly increase memory fragmentation over long-term operation.
|
|
||||||
|
|
||||||
```c
|
|
||||||
// 1. Create region_buffer by specifying a memory block
|
|
||||||
char buffer[512];
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
int err = xdb_region_buffer_init(®ion, buffer, sizeof(buffer));
|
|
||||||
if (err != 0) {
|
|
||||||
// Initialization failed
|
|
||||||
printf("failed to init region buffer width errcode=%d", err);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 2. Create region_buffer by specifying NULL to let it allocate memory as needed automatically
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
int err = xdb_region_buffer_init(®ion, NULL, 0);
|
|
||||||
if (err != 0) {
|
|
||||||
// Initialization failed
|
|
||||||
printf("failed to init region buffer width errcode=%d", err);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
// Note: After each query call, you must manually call the function to free memory.
|
|
||||||
// The search function will report an error if used with uncleaned region info.
|
|
||||||
xdb_region_buffer_free(®ion);
|
|
||||||
```
|
|
||||||
|
|
||||||
# Compiling the Test Program
|
|
||||||
|
|
||||||
Compile and obtain the `xdb_searcher` executable as follows:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# cd to the c binding root directory
|
|
||||||
➜ c git:(master) ✗ make
|
|
||||||
gcc -std=c99 -Wall -O2 -I./ xdb_util.c xdb_searcher.c main.c -o xdb_searcher
|
|
||||||
gcc -std=c99 -Wall -O2 -I./ xdb_util.c test_util.c -o test_util
|
|
||||||
```
|
|
||||||
|
|
||||||
# Query Testing
|
|
||||||
|
|
||||||
Test queries against xdb via the `xdb_searcher search` command:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher search
|
|
||||||
./xdb_searcher search [command options]
|
|
||||||
options:
|
|
||||||
--db string ip2region binary xdb file path
|
|
||||||
--cache-policy string cache policy: file/vectorIndex/content
|
|
||||||
```
|
|
||||||
|
|
||||||
Example: performing IPv4 query testing using the default data/ip2region_v4.xdb:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher search --db=../../data/ip2region_v4.xdb
|
|
||||||
ip2region xdb searcher test program
|
|
||||||
source xdb: ../../data/ip2region_v4.xdb (IPv4, vectorIndex)
|
|
||||||
type 'quit' to exit
|
|
||||||
ip2region>> 1.2.3.4
|
|
||||||
{region: Australia|Queensland|Brisbane|0|AU, io_count: 5, took: 39 μs}
|
|
||||||
ip2region>> 120.229.45.2
|
|
||||||
{region: 中国|广东省|深圳市|移动|CN, io_count: 3, took: 13 μs}
|
|
||||||
```
|
|
||||||
|
|
||||||
Example: performing IPv6 query testing using the default data/ip2region_v6.xdb:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher search --db=../../data/ip2region_v6.xdb
|
|
||||||
ip2region xdb searcher test program
|
|
||||||
source xdb: ../../data/ip2region_v6.xdb (IPv6, vectorIndex)
|
|
||||||
type 'quit' to exit
|
|
||||||
ip2region>> ::
|
|
||||||
{region: , io_count: 1, took: 38 μs}
|
|
||||||
ip2region>> 2604:bc80:8001:11a4:ffff:ffff:ffff:ffff
|
|
||||||
{region: United States|Florida|Miami|velia.net Internetdienste GmbH|US, io_count: 14, took: 76 μs}
|
|
||||||
ip2region>> 240e:3b7:3272:d8d0:db09:c067:8d59:539e
|
|
||||||
{region: 中国|广东省|深圳市|电信|CN, io_count: 8, took: 42 μs}
|
|
||||||
```
|
|
||||||
|
|
||||||
Enter an IP to perform a query; enter `quit` to exit the test program. You can also set `cache-policy` to file/vectorIndex/content respectively to test the efficiency of the three different cache implementations.
|
|
||||||
|
|
||||||
# bench Testing
|
|
||||||
|
|
||||||
Perform bench testing via the `xdb_searcher bench` command. This ensures there are no errors in the query program and the `xdb` file, while also providing average query performance through a large number of queries:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher bench
|
|
||||||
./xdb_searcher bench [command options]
|
|
||||||
options:
|
|
||||||
--db string ip2region binary xdb file path
|
|
||||||
--src string source ip text file path
|
|
||||||
--cache-policy string cache policy: file/vectorIndex/content
|
|
||||||
```
|
|
||||||
|
|
||||||
Example: performing IPv4 bench testing via the default data/ip2region_v4.xdb and data/ipv4_source.txt:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher bench --db=../../data/ip2region_v4.xdb --src=../../data/ipv4_source.txt
|
|
||||||
Bench finished, {cache_policy: vectorIndex, total: 1367686, took: 7.640s, cost: 5 μs/op}
|
|
||||||
```
|
|
||||||
|
|
||||||
Example: performing IPv6 bench testing via the default data/ip2region_v6.xdb and data/ipv6_source.txt:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher bench --db=../../data/ip2region_v6.xdb --src=../../data/ipv6_source.txt
|
|
||||||
Bench finished, {cache_policy: vectorIndex, total: 34159862, took: 857.750s, cost: 24 μs/op}
|
|
||||||
```
|
|
||||||
|
|
||||||
You can set the `cache-policy` parameter to test the efficiency of different cache mechanisms (file/vectorIndex/content). @Note: Please ensure that the `src` file used for benching is the same source file used to generate the corresponding `xdb` file.
|
|
||||||
|
|
@ -1,357 +0,0 @@
|
||||||
:globe_with_meridians: [中文简体](README_zh.md) | [English](README.md)
|
|
||||||
|
|
||||||
# ip2region c 查询客户端
|
|
||||||
|
|
||||||
|
|
||||||
# 使用方式
|
|
||||||
|
|
||||||
### 关于查询 API
|
|
||||||
查询 API 的原型如下：
|
|
||||||
```c
|
|
||||||
// 通过字符串 IP 进行查询
|
|
||||||
int xdb_search_by_string(xdb_searcher_t *, const string_ip_t *, xdb_region_buffer_t *);
|
|
||||||
// 通过 xdb_parse_ip 返回的二进制 IP 进行查询
|
|
||||||
int xdb_search(xdb_searcher_t *, const bytes_ip_t *, int, xdb_region_buffer_t *);
|
|
||||||
```
|
|
||||||
如果查询失败将会返回非 `0` 的错误代码，如果查询成功 xdb_region_buffer_t 可以获取到字符串的 `region` 信息，如果输入的 IP 找不到相关的信息，xdb_region_buffer_t 将会得到一个空的字符串 `""`。
|
|
||||||
|
|
||||||
### 关于 IPv4 和 IPv6
|
|
||||||
该 xdb 查询客户端实现同时支持对 IPv4 和 IPv6 的查询，使用方式如下：
|
|
||||||
```c
|
|
||||||
#include "xdb_api.h";
|
|
||||||
|
|
||||||
// 如果是 IPv4: 设置 xdb 路径为 v4 的 xdb 文件，IP版本指定为 IPv4
|
|
||||||
const char *db_path = "../../data/ip2region_v4.xdb"; // 或者你的 ipv4 xdb 的路径
|
|
||||||
xdb_version_t *version = XDB_IPv4;
|
|
||||||
|
|
||||||
// 如果是 IPv6: 设置 xdb 路径为 v6 的 xdb 文件，IP版本指定为 IPv6
|
|
||||||
const char *db_path = "../../data/ip2region_v6.xdb"; // 或者你的 ipv6 xdb 路径
|
|
||||||
xdb_version_t *version = XDB_IPv6;
|
|
||||||
|
|
||||||
// db_path 指定的 xdb 的 IP 版本必须和 version 指定的一致，不然查询执行的时候会报错
|
|
||||||
// 备注：以下演示直接使用 db_path 和 version 变量
|
|
||||||
```
|
|
||||||
|
|
||||||
### XDB 文件验证
|
|
||||||
建议您主动去验证 xdb 文件的适用性，因为后期的一些新功能可能会导致目前的 Searcher 版本无法适用你使用的 xdb 文件，验证可以避免运行过程中的一些不可预测的错误。 你不需要每次都去验证，例如在服务启动的时候，或者手动调用命令验证确认版本匹配即可，不要在每次创建的 Searcher 的时候运行验证，这样会影响查询的响应速度，尤其是高并发的使用场景。
|
|
||||||
```c
|
|
||||||
#include "xdb_api.h";
|
|
||||||
|
|
||||||
int errcode = xdb_verify(db_path);
|
|
||||||
if ($err != 0) {
|
|
||||||
// 适用性验证失败！！！
|
|
||||||
// 当前查询客户端实现不适用于 db_path 指定的 xdb 文件的查询.
|
|
||||||
// 应该停止启动服务，使用合适的 xdb 文件或者升级到适合 db_path 的 Searcher 实现。
|
|
||||||
printf("failed to verify xdb file `%s`, errcode: %d\n", db_path, errcode);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 验证通过，当前使用的 Searcher 可以安全的用于对 dbPath 指向的 xdb 的查询操作
|
|
||||||
```
|
|
||||||
|
|
||||||
### 完全基于文件的查询
|
|
||||||
|
|
||||||
```c
|
|
||||||
#include <stdio.h>
|
|
||||||
#include "xdb_api.h"
|
|
||||||
|
|
||||||
int main(int argc, char *argv[]) {
|
|
||||||
xdb_searcher_t searcher;
|
|
||||||
char region_buffer[512] = {'\0'};
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
|
|
||||||
// 使用栈空间的 region_buffer 初始化 region_buffer_t
|
|
||||||
int err = xdb_region_buffer_init(®ion, region_buffer, sizeof(region_buffer));
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the region buffer with errcode=%d\n", err);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 在服务启动的时候初始化 winsock，不需要重复调用，只需要在 windows 系统下调用
|
|
||||||
err = xdb_init_winsock();
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the winsock with errno=%d\n", err);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 1、从 db_path 初始化 xdb 查询对象.
|
|
||||||
// @Note: 使用顶部描述的 db_path 和 version 来创建 searcher
|
|
||||||
err = xdb_new_with_file_only(version, &searcher, db_path);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to create xdb searcher from `%s` with errno=%d\n", db_path, err);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 2、调用 search API 查询，IPv4 和 IPv6 都支持.
|
|
||||||
const char *ip_string = "1.2.3.4";
|
|
||||||
// ip_string = "240e:3b7:3272:d8d0:db09:c067:8d59:539e"; // IPv6
|
|
||||||
|
|
||||||
long cost_time = 0, s_time = xdb_now();
|
|
||||||
err = xdb_search_by_string(&searcher, ip_string, ®ion);
|
|
||||||
cost_time = (int) (xdb_now() - s_time);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed search(%s) with errno=%d\n", ip_string, err);
|
|
||||||
} else {
|
|
||||||
printf("{region: %s, took: %d μs}", region.value, cost_time);
|
|
||||||
}
|
|
||||||
|
|
||||||
// 清理 region 信息的内存资源，每次 search 之后都得调用
|
|
||||||
xdb_region_buffer_free(®ion);
|
|
||||||
|
|
||||||
// 备注：并发使用，每一个线程需要单独定义并且初始化一个 searcher 查询对象。
|
|
||||||
|
|
||||||
// 3、关闭 xdb 查询器
|
|
||||||
xdb_close(&searcher);
|
|
||||||
xdb_clean_winsock(); // windows 下调用
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 缓存 `VectorIndex` 索引
|
|
||||||
|
|
||||||
我们可以提前从 xdb 文件中加载出来 VectorIndex 数据，然后全局缓存，每次创建 Searcher 对象的时候使用全局的 VectorIndex 缓存可以减少一次固定的 IO 操作，从而加速查询，减少 IO 压力。
|
|
||||||
```c
|
|
||||||
#include <stdio.h>
|
|
||||||
#include "xdb_api.h"
|
|
||||||
|
|
||||||
int main(int argc, char *argv[]) {
|
|
||||||
xdb_vector_index_t *v_index;
|
|
||||||
xdb_searcher_t searcher;
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
|
|
||||||
// 使用 NULL 初始化 region_buffer，让其自动管理内存的分配
|
|
||||||
int err = xdb_region_buffer_init(®ion, NULL, 0);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the region buffer with errcode=%d\n", err);
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 在服务启动的时候初始化 winsock，不需要重复调用，只需要在 windows 系统下调用
|
|
||||||
err = xdb_init_winsock();
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the winsock with errno=%d\n", err);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 1、从顶部描述的 db_path 加载 VectorIndex 索引。
|
|
||||||
// 得到 v_index 做成全局缓存，便于后续反复使用。
|
|
||||||
// 注意：v_index 不需要每次都加载，建议在服务启动的时候加载一次，然后做成全局资源。
|
|
||||||
v_index = xdb_load_vector_index_from_file(db_path);
|
|
||||||
if (v_index == NULL) {
|
|
||||||
printf("failed to load vector index from `%s`\n", db_path);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 2、使用全局的 VectorIndex 变量创建带 VectorIndex 缓存的 xdb 查询对象.
|
|
||||||
// @Note: 使用顶部描述的 db_path 和 version 来创建 searcher
|
|
||||||
err = xdb_new_with_vector_index(version, &searcher, db_path, v_index);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to create vector index cached searcher with errcode=%d\n", err);
|
|
||||||
return 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
// 3、调用 search API 查询，IPv4 和 IPv6 都支持
|
|
||||||
const char *ip_string = "1.2.3.4";
|
|
||||||
// ip_string = "240e:3b7:3272:d8d0:db09:c067:8d59:539e"; // IPv6
|
|
||||||
|
|
||||||
long cost_time = 0, s_time = xdb_now();
|
|
||||||
err = xdb_search_by_string(&searcher, ip_string, ®ion);
|
|
||||||
cost_time = (int) (xdb_now() - s_time);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed search(%s) with errno=%d\n", ip_string, err);
|
|
||||||
} else {
|
|
||||||
printf("{region: %s, took: %d μs}", region.value, cost_time);
|
|
||||||
}
|
|
||||||
|
|
||||||
// 清理 region 信息的内存资源，每次 search 之后都得调用
|
|
||||||
xdb_region_buffer_free(®ion);
|
|
||||||
|
|
||||||
|
|
||||||
// 备注：并发使用，每一个线程需要单独定义并且初始化一个 searcher 查询对象。
|
|
||||||
|
|
||||||
// 4、关闭 xdb 查询器，如果是要关闭服务，也需要释放 v_index 的内存。
|
|
||||||
xdb_close(&searcher);
|
|
||||||
xdb_close_vector_index(v_index);
|
|
||||||
xdb_clean_winsock();
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 缓存整个 `xdb` 文件
|
|
||||||
|
|
||||||
我们也可以预先加载整个 xdb 文件到内存，然后基于这个数据创建查询对象来实现完全基于内存的查询，类似之前的 memory search。
|
|
||||||
```c
|
|
||||||
#include <stdio.h>
|
|
||||||
#include "xdb_api.h"
|
|
||||||
|
|
||||||
int main(int argc, char *argv[]) {
|
|
||||||
xdb_content_t *c_buffer;
|
|
||||||
xdb_searcher_t searcher;
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
|
|
||||||
// 使用 NULL 初始化 region_buffer，让其自动管理内存的分配
|
|
||||||
int err = xdb_region_buffer_init(®ion, NULL, 0);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the region buffer with errcode=%d\n", err);
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
// 在服务启动的时候初始化 winsock，不需要重复调用，只需要在 windows 系统下调用
|
|
||||||
err = xdb_init_winsock();
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the winsock with errno=%d\n", err);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 1、从 顶部描述的 db_path 加载整个 xdb 的数据。
|
|
||||||
c_buffer = xdb_load_content_from_file(db_path);
|
|
||||||
if (v_index == NULL) {
|
|
||||||
printf("failed to load xdb content from `%s`\n", db_path);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 2、使用全局的 c_buffer 变量创建一个完全基于内存的 xdb 查询对象.
|
|
||||||
// @Note: 使用顶部描述的 version 来创建 searcher.
|
|
||||||
err = xdb_new_with_buffer(version, &searcher, c_buffer);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to create content cached searcher with errcode=%d\n", err);
|
|
||||||
return 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 3、调用 search API 查询，IPv4 和 IPv6 都支持
|
|
||||||
const char *ip_string = "1.2.3.4";
|
|
||||||
// ip_string = "240e:3b7:3272:d8d0:db09:c067:8d59:539e"; // IPv6
|
|
||||||
|
|
||||||
long cost_time = 0, s_time = xdb_now();
|
|
||||||
err = xdb_search_by_string(&searcher, ip_string, ®ion);
|
|
||||||
cost_time = (int) (xdb_now() - s_time);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed search(%s) with errno=%d\n", ip_string, err);
|
|
||||||
} else {
|
|
||||||
printf("{region: %s, took: %d μs}", region.value, cost_time);
|
|
||||||
}
|
|
||||||
|
|
||||||
// 清理 region 信息的内存资源，每次 search 之后都得调用
|
|
||||||
xdb_region_buffer_free(®ion);
|
|
||||||
|
|
||||||
|
|
||||||
// 备注：并发使用，使用这种方式创建的 xdb 查询对象可以安全用于并发。
|
|
||||||
// 建议在服务启动的时候创建好，然后一直安全并发使用，直到服务关闭。
|
|
||||||
|
|
||||||
// 4、关闭 xdb 查询器，关闭服务的时候需要释放 c_buffer 的内存。
|
|
||||||
xdb_close(&searcher);
|
|
||||||
xdb_close_content(c_buffer);
|
|
||||||
xdb_clean_winsock();
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 关于定位信息的存储
|
|
||||||
在旧版本的实现中，search相关的函数都是依靠指定一个 `region_buffer` 内存来用于存储地域信息，这种方式还是有很大的局限性。
|
|
||||||
新的实现提供了一个 `xdb_region_buffer_t` 对象来管理这些内存的分配，你依然可以指定一个固定的 `region_buffer` 来创建 region 的内存管理，这个情况适合当你的地域信息的最大长度是可知的，这种方式可以减少运行过程中内存的碎片堆积。如果地域信息的长度不确定或者你的程序不适合提前分配一块内存来管理，你可以通过指定 NULL 的方式来初始化 `xdb_region_buffer_t`，这样对象会自动管理内存的分配，也适合任意长度的地域信息的存储，不过这种方式在长期的运行过程中肯定会增加内存碎片的堆积。
|
|
||||||
```c
|
|
||||||
// 1, 通过指定一块内存来创建 region_buffer
|
|
||||||
char buffer[512];
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
int err = xdb_region_buffer_init(®ion, buffer, sizeof(buffer));
|
|
||||||
if (err != 0) {
|
|
||||||
// 初始化失败
|
|
||||||
printf("failed to init region buffer width errcode=%d", err);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 2，通过指定 NULL 来创建 region_buffer，让其自动按需分配内存
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
int err = xdb_region_buffer_init(®ion, NULL, 0);
|
|
||||||
if (err != 0) {
|
|
||||||
// 初始化失败
|
|
||||||
printf("failed to init region buffer width errcode=%d", err);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
// 备注：在每次调用 search 完成 IP 定位信息的查询后，你需要手动调用函数来释放内存 .
|
|
||||||
// search 函数使用未经清理的 region 信息会报错。
|
|
||||||
xdb_region_buffer_free(®ion);
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
# 测试程序编译
|
|
||||||
|
|
||||||
通过如下方式编译得到 xdb_searcher 可执行程序：
|
|
||||||
```bash
|
|
||||||
# cd 到 c binding 根目录
|
|
||||||
➜ c git:(master) ✗ make
|
|
||||||
gcc -std=c99 -Wall -O2 -I./ xdb_util.c xdb_searcher.c main.c -o xdb_searcher
|
|
||||||
gcc -std=c99 -Wall -O2 -I./ xdb_util.c test_util.c -o test_util
|
|
||||||
```
|
|
||||||
|
|
||||||
|
|
||||||
# 查询测试
|
|
||||||
|
|
||||||
通过 `xdb_searcher search` 命令来测试对 xdb 的查询：
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher search
|
|
||||||
./xdb_searcher search [command options]
|
|
||||||
options:
|
|
||||||
--db string ip2region binary xdb file path
|
|
||||||
--cache-policy string cache policy: file/vectorIndex/content
|
|
||||||
```
|
|
||||||
|
|
||||||
例如：使用默认的 data/ip2region_v4.xdb 进行 IPv4 查询测试：
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher search --db=../../data/ip2region_v4.xdb
|
|
||||||
ip2region xdb searcher test program
|
|
||||||
source xdb: ../../data/ip2region_v4.xdb (IPv4, vectorIndex)
|
|
||||||
type 'quit' to exit
|
|
||||||
ip2region>> 1.2.3.4
|
|
||||||
{region: Australia|Queensland|Brisbane|0|AU, io_count: 5, took: 39 μs}
|
|
||||||
ip2region>> 120.229.45.2
|
|
||||||
{region: 中国|广东省|深圳市|移动|CN, io_count: 3, took: 13 μs}
|
|
||||||
```
|
|
||||||
|
|
||||||
例如：使用默认的 data/ip2region_v6.xdb 进行 IPv6 查询测试：
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher search --db=../../data/ip2region_v6.xdb
|
|
||||||
ip2region xdb searcher test program
|
|
||||||
source xdb: ../../data/ip2region_v6.xdb (IPv6, vectorIndex)
|
|
||||||
type 'quit' to exit
|
|
||||||
ip2region>> ::
|
|
||||||
{region: , io_count: 1, took: 38 μs}
|
|
||||||
ip2region>> 2604:bc80:8001:11a4:ffff:ffff:ffff:ffff
|
|
||||||
{region: United States|Florida|Miami|velia.net Internetdienste GmbH|US, io_count: 14, took: 76 μs}
|
|
||||||
ip2region>> 240e:3b7:3272:d8d0:db09:c067:8d59:539e
|
|
||||||
{region: 中国|广东省|深圳市|电信|CN, io_count: 8, took: 42 μs}
|
|
||||||
```
|
|
||||||
|
|
||||||
输入 ip 即可进行查询，输入 quit 即可退出测试程序。也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同的缓存实现的效率。
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
# bench 测试
|
|
||||||
|
|
||||||
通过 `xdb_searcher bench` 命令来进行 bench 测试，一方面确保查询程序和 `xdb` 文件没有错误，另一方面可以通过大量的查询得到平均的查询性能：
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher bench
|
|
||||||
./xdb_searcher bench [command options]
|
|
||||||
options:
|
|
||||||
--db string ip2region binary xdb file path
|
|
||||||
--src string source ip text file path
|
|
||||||
--cache-policy string cache policy: file/vectorIndex/content
|
|
||||||
```
|
|
||||||
|
|
||||||
例如：通过默认的 data/ip2region_v4.xdb 和 data/ipv4_source.txt 来进行 IPv4 的 bench 测试：
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher bench --db=../../data/ip2region_v4.xdb --src=../../data/ipv4_source.txt
|
|
||||||
Bench finished, {cache_policy: vectorIndex, total: 1367686, took: 7.640s, cost: 5 μs/op}
|
|
||||||
```
|
|
||||||
|
|
||||||
例如：通过默认的 data/ip2region_v6.xdb 和 data/ipv6_source.txt 来进行 IPv6 的 bench 测试：
|
|
||||||
```bash
|
|
||||||
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher bench --db=../../data/ip2region_v6.xdb --src=../../data/ipv6_source.txt
|
|
||||||
Bench finished, {cache_policy: vectorIndex, total: 34159862, took: 857.750s, cost: 24 μs/op}
|
|
||||||
```
|
|
||||||
|
|
||||||
可以设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 不同缓存实现机制的效率。 @Note：请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件相同的源文件。
|
|
||||||
|
|
@ -0,0 +1,193 @@
|
||||||
|
# ip2region xdb c 查询客户端实现
|
||||||
|
|
||||||
|
|
||||||
|
# 使用方式
|
||||||
|
|
||||||
|
### 完全基于文件的查询
|
||||||
|
|
||||||
|
```c
|
||||||
|
#include <stdio.h>
|
||||||
|
#include "xdb_searcher.h"
|
||||||
|
|
||||||
|
int main(int argc, char *argv[]) {
|
||||||
|
char *db_path = "ip2region.xdb file path";
|
||||||
|
xdb_searcher_t searcher;
|
||||||
|
char region_buffer[256], ip_buffer[16], *ip = "1.2.3.4";
|
||||||
|
long s_time;
|
||||||
|
|
||||||
|
// 1、从 db_path 初始化 xdb 查询对象
|
||||||
|
int err = xdb_new_with_file_only(&searcher, db_path);
|
||||||
|
if (err != 0) {
|
||||||
|
printf("failed to create xdb searcher from `%s` with errno=%d\n", db_path, err);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2、调用 search API 查询
|
||||||
|
// 得到的 region 信息会存储到 region_buffer 里面，如果你自定义了数据，请确保给足 buffer 的空间。
|
||||||
|
s_time = xdb_now();
|
||||||
|
err = xdb_search_by_string(&searcher, ip, region_buffer, sizeof(region_buffer));
|
||||||
|
if (err != 0) {
|
||||||
|
printf("failed search(%s) with errno=%d\n", ip, err);
|
||||||
|
} else {
|
||||||
|
printf("{region: %s, took: %d μs}", region_buffer, (int)(xdb_now() - s_time));
|
||||||
|
}
|
||||||
|
|
||||||
|
// 备注：并发使用，没一个线程需要单独定义并且初始化一个 searcher 查询对象。
|
||||||
|
|
||||||
|
// 3、关闭 xdb 查询器
|
||||||
|
xdb_close(&searcher);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 缓存 `VectorIndex` 索引
|
||||||
|
|
||||||
|
我们可以提前从 xdb 文件中加载出来 VectorIndex 数据，然后全局缓存，每次创建 Searcher 对象的时候使用全局的 VectorIndex 缓存可以减少一次固定的 IO 操作，从而加速查询，减少 IO 压力。
|
||||||
|
```c
|
||||||
|
#include <stdio.h>
|
||||||
|
#include "xdb_searcher.h"
|
||||||
|
|
||||||
|
int main(int argc, char *argv[]) {
|
||||||
|
char *db_path = "ip2region.xdb file path";
|
||||||
|
xdb_vector_index_t *v_index;
|
||||||
|
xdb_searcher_t searcher;
|
||||||
|
char region_buffer[256], ip_buffer[16], *ip = "1.2.3.4";
|
||||||
|
long s_time;
|
||||||
|
|
||||||
|
// 1、从 db_path 加载 VectorIndex 索引。
|
||||||
|
// 得到 v_index 做成全局缓存，便于后续反复使用。
|
||||||
|
// 注意：v_index 不需要每次都加载，建议在服务启动的时候加载一次，然后做成全局资源。
|
||||||
|
v_index = xdb_load_vector_index_from_file(db_path);
|
||||||
|
if (v_index == NULL) {
|
||||||
|
printf("failed to load vector index from `%s`\n", db_path);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2、使用全局的 VectorIndex 变量创建带 VectorIndex 缓存的 xdb 查询对象
|
||||||
|
int err = xdb_new_with_vector_index(&searcher, db_path, v_index);
|
||||||
|
if (err != 0) {
|
||||||
|
printf("failed to create vector index cached searcher with errcode=%d\n", err);
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3、调用 search API 查询
|
||||||
|
// 得到的 region 信息会存储到 region_buffer 里面，如果你自定义了数据，请确保给足 buffer 的空间。
|
||||||
|
s_time = xdb_now();
|
||||||
|
err = xdb_search_by_string(&searcher, ip, region_buffer, sizeof(region_buffer));
|
||||||
|
if (err != 0) {
|
||||||
|
printf("failed search(%s) with errno=%d\n", ip, err);
|
||||||
|
} else {
|
||||||
|
printf("{region: %s, took: %d μs}", region_buffer, (int)(xdb_now() - s_time));
|
||||||
|
}
|
||||||
|
|
||||||
|
// 备注：并发使用，没一个线程需要单独定义并且初始化一个 searcher 查询对象。
|
||||||
|
|
||||||
|
// 4、关闭 xdb 查询器，如果是要关闭服务，也需要释放 v_index 的内存。
|
||||||
|
xdb_close(&searcher);
|
||||||
|
xdb_close_vector_index(v_index);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 缓存整个 `xdb` 数据
|
||||||
|
|
||||||
|
我们也可以预先加载整个 ip2region.xdb 的数据到内存，然后基于这个数据创建查询对象来实现完全基于文件的查询，类似之前的 memory search。
|
||||||
|
```c
|
||||||
|
#include <stdio.h>
|
||||||
|
#include "xdb_searcher.h"
|
||||||
|
|
||||||
|
int main(int argc, char *argv[]) {
|
||||||
|
char *db_path = "ip2region.xdb file path";
|
||||||
|
xdb_content_t *c_buffer;
|
||||||
|
xdb_searcher_t searcher;
|
||||||
|
char region_buffer[256], ip_buffer[16], *ip = "1.2.3.4";
|
||||||
|
long s_time;
|
||||||
|
|
||||||
|
// 1、从 db_path 加载整个 xdb 的数据。
|
||||||
|
c_buffer = xdb_load_content_from_file(db_path);
|
||||||
|
if (v_index == NULL) {
|
||||||
|
printf("failed to load xdb content from `%s`\n", db_path);
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 2、使用全局的 c_buffer 变量创建一个完全基于内存的 xdb 查询对象
|
||||||
|
err = xdb_new_with_buffer(&searcher, c_buffer);
|
||||||
|
if (err != 0) {
|
||||||
|
printf("failed to create content cached searcher with errcode=%d\n", err);
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
// 3、调用 search API 查询
|
||||||
|
// 得到的 region 信息会存储到 region_buffer 里面，如果你自定义了数据，请确保给足 buffer 的空间。
|
||||||
|
s_time = xdb_now();
|
||||||
|
err = xdb_search_by_string(&searcher, ip, region_buffer, sizeof(region_buffer));
|
||||||
|
if (err != 0) {
|
||||||
|
printf("failed search(%s) with errno=%d\n", ip, err);
|
||||||
|
} else {
|
||||||
|
printf("{region: %s, took: %d μs}", region_buffer, (int)(xdb_now() - s_time));
|
||||||
|
}
|
||||||
|
|
||||||
|
// 备注：并发使用，使用这种方式创建的 xdb 查询对象可以安全用于并发。
|
||||||
|
// 建议在服务启动的时候创建好，然后一直安全并发使用，直到服务关闭。
|
||||||
|
|
||||||
|
// 4、关闭 xdb 查询器，关闭服务的时候需要释放 c_buffer 的内存。
|
||||||
|
xdb_close(&searcher);
|
||||||
|
xdb_close_content(c_buffer);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
# 测试程序编译
|
||||||
|
|
||||||
|
通过如下方式编译得到 xdb_searcher 可执行程序：
|
||||||
|
```bash
|
||||||
|
# cd 到 c binding 根目录
|
||||||
|
make
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
# 查询测试
|
||||||
|
|
||||||
|
通过 `xdb_searcher search` 命令来测试对 ip2region.xdb 的查询：
|
||||||
|
```bash
|
||||||
|
➜ c git:(c_binding) ✗ ./xdb_searcher search
|
||||||
|
./xdb_searcher search [command options]
|
||||||
|
options:
|
||||||
|
--db string ip2region binary xdb file path
|
||||||
|
--cache-policy string cache policy: file/vectorIndex/content
|
||||||
|
```
|
||||||
|
|
||||||
|
例如：使用默认的 data/ip2region.xdb 进行查询测试：
|
||||||
|
```bash
|
||||||
|
➜ c git:(c_binding) ✗ ./xdb_searcher search --db=../../data/ip2region.xdb --cache-policy=vectorIndex
|
||||||
|
ip2region xdb searcher test program, cache_policy: vectorIndex
|
||||||
|
type 'quit' to exit
|
||||||
|
ip2region>> 1.2.3.4
|
||||||
|
{region: 美国|0|华盛顿|0|谷歌, io_count: 7, took: 13 μs}
|
||||||
|
ip2region>>
|
||||||
|
```
|
||||||
|
|
||||||
|
输入 ip 即可进行查询，输入 quit 即可退出测试程序。也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同的缓存实现的效率。
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
# bench 测试
|
||||||
|
|
||||||
|
通过 `xdb_searcher bench` 命令来进行 bench 测试，一方面确保查询程序和 `xdb` 文件没有错误，另一方面可以通过大量的查询得到评价的查询性能：
|
||||||
|
```bash
|
||||||
|
➜ c git:(c_binding) ✗ ./xdb_searcher bench
|
||||||
|
./xdb_searcher bench [command options]
|
||||||
|
options:
|
||||||
|
--db string ip2region binary xdb file path
|
||||||
|
--src string source ip text file path
|
||||||
|
--cache-policy string cache policy: file/vectorIndex/content
|
||||||
|
```
|
||||||
|
|
||||||
|
例如：通过默认的 data/ip2region.xdb 和 data/ip.merge.txt 来进行 bench 测试：
|
||||||
|
```bash
|
||||||
|
➜ c git:(c_binding) ✗ ./xdb_searcher bench --db=../../data/ip2region.xdb --src=../../data/ip.merge.txt --cache-policy=vectorIndex
|
||||||
|
Bench finished, {cache_policy: vectorIndex, total: 3417955, took: 4.233s, cost: 1 μs/op}
|
||||||
|
```
|
||||||
|
|
||||||
|
可以设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 不同缓存实现机制的效率。 @Note：请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件相同的源文件。
|
||||||
181
binding/c/main.c
181
binding/c/main.c
|
|
@ -7,112 +7,56 @@
|
||||||
// @Date 2022/06/28
|
// @Date 2022/06/28
|
||||||
|
|
||||||
#include "stdio.h"
|
#include "stdio.h"
|
||||||
#include "xdb_api.h"
|
#include "xdb_searcher.h"
|
||||||
|
|
||||||
struct searcher_test_entry {
|
struct searcher_test_entry {
|
||||||
xdb_searcher_t searcher;
|
xdb_searcher_t searcher;
|
||||||
xdb_vector_index_t *v_index;
|
xdb_vector_index_t *v_index;
|
||||||
xdb_content_t *c_buffer;
|
xdb_content_t *c_buffer;
|
||||||
|
|
||||||
// xdb region buffer
|
|
||||||
// char region_buffer[256];
|
|
||||||
xdb_region_buffer_t region;
|
|
||||||
};
|
};
|
||||||
typedef struct searcher_test_entry searcher_test_t;
|
typedef struct searcher_test_entry searcher_test_t;
|
||||||
|
|
||||||
int init_searcher_test(searcher_test_t *test, char *db_path, char *cache_policy) {
|
int init_searcher_test(searcher_test_t *test, char *db_path, char *cache_policy) {
|
||||||
int err, errcode = 0;
|
int err;
|
||||||
FILE *handle = fopen(db_path, "rb");
|
|
||||||
if (handle == NULL) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// auto detect the version from the xdb header
|
|
||||||
xdb_header_t *header = xdb_load_header(handle);
|
|
||||||
if (header == NULL) {
|
|
||||||
printf("failed to load header from `%s`\n", db_path);
|
|
||||||
errcode = 1;
|
|
||||||
goto defer;
|
|
||||||
}
|
|
||||||
|
|
||||||
// verify the current xdb
|
|
||||||
err = xdb_verify_from_header(handle, header);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to verify xdb file `%s` with errno=%d\n", db_path, err);
|
|
||||||
errcode = 2;
|
|
||||||
goto defer;
|
|
||||||
}
|
|
||||||
|
|
||||||
xdb_version_t *version = xdb_version_from_header(header);
|
|
||||||
if (version == NULL) {
|
|
||||||
printf("failed to load version from header\n");
|
|
||||||
errcode = 3;
|
|
||||||
goto defer;
|
|
||||||
}
|
|
||||||
|
|
||||||
test->v_index = NULL;
|
test->v_index = NULL;
|
||||||
test->c_buffer = NULL;
|
test->c_buffer = NULL;
|
||||||
|
|
||||||
if (strcmp(cache_policy, "file") == 0) {
|
if (strcmp(cache_policy, "file") == 0) {
|
||||||
err = xdb_new_with_file_only(version, &test->searcher, db_path);
|
err = xdb_new_with_file_only(&test->searcher, db_path);
|
||||||
if (err != 0) {
|
if (err != 0) {
|
||||||
printf("failed to create searcher with errcode=%d\n", err);
|
printf("failed to create searcher with errcode=%d\n", err);
|
||||||
errcode = 4;
|
return 1;
|
||||||
goto defer;
|
|
||||||
}
|
}
|
||||||
} else if (strcmp(cache_policy, "vectorIndex") == 0) {
|
} else if (strcmp(cache_policy, "vectorIndex") == 0) {
|
||||||
test->v_index = xdb_load_vector_index_from_file(db_path);
|
test->v_index = xdb_load_vector_index_from_file(db_path);
|
||||||
if (test->v_index == NULL) {
|
if (test->v_index == NULL) {
|
||||||
printf("failed to load vector index from `%s`\n", db_path);
|
printf("failed to load vector index from `%s`\n", db_path);
|
||||||
errcode = 4;
|
return 2;
|
||||||
goto defer;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
err = xdb_new_with_vector_index(version, &test->searcher, db_path, test->v_index);
|
err = xdb_new_with_vector_index(&test->searcher, db_path, test->v_index);
|
||||||
if (err != 0) {
|
if (err != 0) {
|
||||||
printf("failed to create vector index cached searcher with errcode=%d\n", err);
|
printf("failed to create vector index cached searcher with errcode=%d\n", err);
|
||||||
errcode = 5;
|
return 3;
|
||||||
goto defer;
|
|
||||||
}
|
}
|
||||||
} else if (strcmp(cache_policy, "content") == 0) {
|
} else if (strcmp(cache_policy, "content") == 0) {
|
||||||
test->c_buffer = xdb_load_content_from_file(db_path);
|
test->c_buffer = xdb_load_content_from_file(db_path);
|
||||||
if (test->c_buffer == NULL) {
|
if (test->c_buffer == NULL) {
|
||||||
printf("failed to load xdb content from `%s`\n", db_path);
|
printf("failed to load xdb content from `%s`\n", db_path);
|
||||||
errcode = 4;
|
return 4;
|
||||||
goto defer;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
err = xdb_new_with_buffer(version, &test->searcher, test->c_buffer);
|
err = xdb_new_with_buffer(&test->searcher, test->c_buffer);
|
||||||
if (err != 0) {
|
if (err != 0) {
|
||||||
printf("failed to create content cached searcher with errcode=%d\n", err);
|
printf("failed to create content cached searcher with errcode=%d\n", err);
|
||||||
errcode = 5;
|
return 5;
|
||||||
goto defer;
|
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
printf("invalid cache policy `%s`, options: file/vectorIndex/content\n", cache_policy);
|
printf("invalid cache policy `%s`, options: file/vectorIndex/content\n", cache_policy);
|
||||||
errcode = 6;
|
return 6;
|
||||||
goto defer;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// init the region buffer
|
return 0;
|
||||||
// err = xdb_region_buffer_init(&test->region, test->region_buffer, sizeof(test->region_buffer));
|
|
||||||
err = xdb_region_buffer_init(&test->region, NULL, 0);
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the region buffer with err=%d\n", err);
|
|
||||||
errcode = 7;
|
|
||||||
goto defer;
|
|
||||||
}
|
|
||||||
|
|
||||||
defer:
|
|
||||||
if (header != NULL) {
|
|
||||||
xdb_free_header(header);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (handle != NULL) {
|
|
||||||
fclose(handle);
|
|
||||||
}
|
|
||||||
|
|
||||||
return errcode;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
void destroy_searcher_test(searcher_test_t *test) {
|
void destroy_searcher_test(searcher_test_t *test) {
|
||||||
|
|
@ -120,13 +64,13 @@ void destroy_searcher_test(searcher_test_t *test) {
|
||||||
|
|
||||||
// check and free the vector index
|
// check and free the vector index
|
||||||
if (test->v_index != NULL) {
|
if (test->v_index != NULL) {
|
||||||
xdb_free_vector_index(test->v_index);
|
xdb_close_vector_index(test->v_index);
|
||||||
test->v_index = NULL;
|
test->v_index = NULL;
|
||||||
}
|
}
|
||||||
|
|
||||||
// check and free the content buffer
|
// check and free the content buffer
|
||||||
if (test->c_buffer != NULL) {
|
if (test->c_buffer != NULL) {
|
||||||
xdb_free_content(test->c_buffer);
|
xdb_close_content(test->c_buffer);
|
||||||
test->c_buffer = NULL;
|
test->c_buffer = NULL;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -163,12 +107,8 @@ void test_search(int argc, char *argv[]) {
|
||||||
|
|
||||||
// for search
|
// for search
|
||||||
long s_time, c_time;
|
long s_time, c_time;
|
||||||
char line[512] = {'\0'};
|
unsigned int ip;
|
||||||
|
char line[512] = {'\0'}, region[512] = {'\0'};
|
||||||
// ip parse
|
|
||||||
xdb_version_t *version;
|
|
||||||
bytes_ip_t ip_bytes[16] = {'\0'};
|
|
||||||
|
|
||||||
searcher_test_t test;
|
searcher_test_t test;
|
||||||
|
|
||||||
for (i = 2; i < argc; i++) {
|
for (i = 2; i < argc; i++) {
|
||||||
|
|
@ -212,13 +152,6 @@ void test_search(int argc, char *argv[]) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// init the win sock
|
|
||||||
err = xdb_init_winsock();
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the winsock with errno=%d\n", err);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// printf("db_file=%s, cache_policy=%s\n", db_file, cache_policy);
|
// printf("db_file=%s, cache_policy=%s\n", db_file, cache_policy);
|
||||||
err = init_searcher_test(&test, db_file, cache_policy);
|
err = init_searcher_test(&test, db_file, cache_policy);
|
||||||
if (err != 0) {
|
if (err != 0) {
|
||||||
|
|
@ -226,9 +159,8 @@ void test_search(int argc, char *argv[]) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
printf("ip2region xdb searcher test program\n"
|
printf("ip2region xdb searcher test program, "
|
||||||
"source xdb: %s (%s, %s)\n"
|
"cache_policy: %s\ntype 'quit' to exit\n", cache_policy);
|
||||||
"type 'quit' to exit\n", db_file, xdb_get_version(&test.searcher)->name, cache_policy);
|
|
||||||
while ( 1 ) {
|
while ( 1 ) {
|
||||||
printf("ip2region>> ");
|
printf("ip2region>> ");
|
||||||
get_line(stdin, line);
|
get_line(stdin, line);
|
||||||
|
|
@ -236,31 +168,26 @@ void test_search(int argc, char *argv[]) {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (strcmp(line, "quit") == 0 ) {
|
if (strcasecmp( line, "quit") == 0 ) {
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
version = xdb_parse_ip(line, ip_bytes, sizeof(ip_bytes));
|
if (xdb_check_ip(line, &ip) != 0) {
|
||||||
if (version == NULL) {
|
|
||||||
printf("invalid ip address `%s`\n", line);
|
printf("invalid ip address `%s`\n", line);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
|
||||||
s_time = xdb_now();
|
s_time = xdb_now();
|
||||||
err = xdb_search(&test.searcher, ip_bytes, version->bytes, &test.region);
|
err = xdb_search(&test.searcher, ip, region, sizeof(region));
|
||||||
if (err != 0) {
|
if (err != 0) {
|
||||||
printf("{err: %d, io_count: %d}\n", err, xdb_get_io_count(&test.searcher));
|
printf("{err: %d, io_count: %d}\n", err, xdb_get_io_count(&test.searcher));
|
||||||
} else {
|
} else {
|
||||||
c_time = xdb_now() - s_time;
|
c_time = xdb_now() - s_time;
|
||||||
printf("{region: %s, io_count: %d, took: %ld μs}\n", test.region.value, xdb_get_io_count(&test.searcher), c_time);
|
printf("{region: %s, io_count: %d, took: %ld μs}\n", region, xdb_get_io_count(&test.searcher), c_time);
|
||||||
}
|
}
|
||||||
|
|
||||||
// free the region
|
|
||||||
xdb_region_buffer_free(&test.region);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
destroy_searcher_test(&test);
|
destroy_searcher_test(&test);
|
||||||
xdb_clean_winsock();
|
|
||||||
printf("searcher test program exited, thanks for trying\n");
|
printf("searcher test program exited, thanks for trying\n");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -270,18 +197,13 @@ void test_bench(int argc, char *argv[]) {
|
||||||
char db_file[256] = {'\0'}, src_file[256] = {'\0'}, cache_policy[16] = {"vectorIndex"};
|
char db_file[256] = {'\0'}, src_file[256] = {'\0'}, cache_policy[16] = {"vectorIndex"};
|
||||||
|
|
||||||
FILE *handle;
|
FILE *handle;
|
||||||
char line[1024] = {'\0'}, sip_str[INET6_ADDRSTRLEN+1] = {'\0'}, eip_str[INET6_ADDRSTRLEN+1] = {'\0'};
|
char line[1024] = {'\0'}, sip_str[16] = {'\0'}, eip_str[16] = {'\0'};
|
||||||
char src_region[512] = {'\0'};
|
char src_region[512] = {'\0'}, region_buffer[512] = {'\0'};
|
||||||
|
unsigned int sip, eip, mip, ip_list[5];
|
||||||
int count = 0, took;
|
int count = 0, took;
|
||||||
long s_time, t_time, c_time = 0;
|
long s_time, t_time, c_time = 0;
|
||||||
|
|
||||||
// ip parse
|
|
||||||
xdb_version_t *s_version, *e_version;
|
|
||||||
bytes_ip_t sip_bytes[16] = {'\0'}, eip_bytes[16] = {'\0'};
|
|
||||||
string_ip_t ip_string[INET6_ADDRSTRLEN] = {'\0'};
|
|
||||||
bytes_ip_t *ip_list[2];
|
|
||||||
|
|
||||||
searcher_test_t test;
|
searcher_test_t test;
|
||||||
|
|
||||||
for (i = 2; i < argc; i++) {
|
for (i = 2; i < argc; i++) {
|
||||||
r = argv[i];
|
r = argv[i];
|
||||||
if (strlen(r) < 5) {
|
if (strlen(r) < 5) {
|
||||||
|
|
@ -324,13 +246,6 @@ void test_bench(int argc, char *argv[]) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// init the win sock
|
|
||||||
err = xdb_init_winsock();
|
|
||||||
if (err != 0) {
|
|
||||||
printf("failed to init the winsock with errno=%d\n", err);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// printf("db_file=%s, src_file=%s, cache_policy=%s\n", db_file, src_file, cache_policy);
|
// printf("db_file=%s, src_file=%s, cache_policy=%s\n", db_file, src_file, cache_policy);
|
||||||
s_time = xdb_now();
|
s_time = xdb_now();
|
||||||
err = init_searcher_test(&test, db_file, cache_policy);
|
err = init_searcher_test(&test, db_file, cache_policy);
|
||||||
|
|
@ -347,63 +262,57 @@ void test_bench(int argc, char *argv[]) {
|
||||||
}
|
}
|
||||||
|
|
||||||
while(fgets(line, sizeof(line), handle) != NULL) {
|
while(fgets(line, sizeof(line), handle) != NULL) {
|
||||||
n = sscanf(line, "%46[^|]|%46[^|]|%511[^\n]", sip_str, eip_str, src_region);
|
n = sscanf(line, "%15[^|]|%15[^|]|%511[^\n]", sip_str, eip_str, src_region);
|
||||||
if (n != 3) {
|
if (n != 3) {
|
||||||
printf("invalid ip segment line `%s`\n", line);
|
printf("invalid ip segment line `%s`\n", line);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
s_version = xdb_parse_ip(sip_str, sip_bytes, sizeof(sip_bytes));
|
if (xdb_check_ip(sip_str, &sip) != 0) {
|
||||||
if (s_version == NULL) {
|
|
||||||
printf("invalid start ip `%s`\n", sip_str);
|
printf("invalid start ip `%s`\n", sip_str);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
e_version = xdb_parse_ip(eip_str, eip_bytes, sizeof(eip_bytes));
|
if (xdb_check_ip(eip_str, &eip) != 0) {
|
||||||
if (e_version == NULL) {
|
|
||||||
printf("invalid end ip `%s`\n", sip_str);
|
printf("invalid end ip `%s`\n", sip_str);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (s_version->id != e_version->id) {
|
if (sip > eip) {
|
||||||
printf("start ip and end ip version not match for line `%s`\n", line);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (xdb_ip_sub_compare(sip_bytes, s_version->bytes, (string_ip_t *) eip_bytes, 0) > 0) {
|
|
||||||
printf("start ip(%s) should not be greater than end ip(%s)\n", sip_str, eip_str);
|
printf("start ip(%s) should not be greater than end ip(%s)\n", sip_str, eip_str);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
ip_list[0] = sip_bytes;
|
mip = xdb_mip(sip, eip);
|
||||||
ip_list[1] = eip_bytes;
|
ip_list[0] = sip;
|
||||||
for (i = 0; i < 2; i++) {
|
ip_list[1] = xdb_mip(sip, mip);
|
||||||
|
ip_list[2] = mip;
|
||||||
|
ip_list[3] = xdb_mip(mip, eip);
|
||||||
|
ip_list[4] = eip;
|
||||||
|
for (i = 0; i < 5; i++) {
|
||||||
t_time = xdb_now();
|
t_time = xdb_now();
|
||||||
err = xdb_search(&test.searcher, ip_list[i], s_version->bytes, &test.region);
|
err = xdb_search(&test.searcher, ip_list[i], region_buffer, sizeof(region_buffer));
|
||||||
c_time += xdb_now() - t_time;
|
|
||||||
if (err != 0) {
|
if (err != 0) {
|
||||||
xdb_ip_to_string(ip_list[i], s_version->bytes, ip_string, sizeof(ip_string));
|
xdb_long2ip(ip_list[i], sip_str);
|
||||||
printf("failed to search ip `%s` with errno=%d\n", ip_string, err);
|
printf("failed to search ip `%s` with errno=%d\n", sip_str, err);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
c_time += xdb_now() - t_time;
|
||||||
|
|
||||||
// check the region info
|
// check the region info
|
||||||
if (strcmp(test.region.value, src_region) != 0) {
|
if (strcmp(region_buffer, src_region) != 0) {
|
||||||
xdb_ip_to_string(ip_list[i], s_version->bytes, ip_string, sizeof(ip_string));
|
xdb_long2ip(ip_list[i], sip_str);
|
||||||
printf("failed to search(%s) with (%s != %s)\n", ip_string, test.region.value, src_region);
|
printf("failed to search(%s) with (%s != %s)\n", sip_str, region_buffer, src_region);
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
// free the region buffer
|
|
||||||
xdb_region_buffer_free(&test.region);
|
|
||||||
|
|
||||||
count++;
|
count++;
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
took = xdb_now() - s_time;
|
took = xdb_now() - s_time;
|
||||||
destroy_searcher_test(&test);
|
destroy_searcher_test(&test);
|
||||||
xdb_clean_winsock();
|
|
||||||
fclose(handle);
|
fclose(handle);
|
||||||
printf("Bench finished, {cache_policy: %s, total: %d, took: %.3fs, cost: %d μs/op}\n",
|
printf("Bench finished, {cache_policy: %s, total: %d, took: %.3fs, cost: %d μs/op}\n",
|
||||||
cache_policy, count, took/1e6, count == 0 ? 0 : (int)(c_time/count));
|
cache_policy, count, took/1e6, count == 0 ? 0 : (int)(c_time/count));
|
||||||
|
|
|
||||||
|
|
@ -1,213 +0,0 @@
|
||||||
// Copyright 2022 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
|
|
||||||
// ---
|
|
||||||
// @Author Lion <chenxin619315@gmail.com>
|
|
||||||
// @Date 2022/06/27
|
|
||||||
|
|
||||||
#include "stdio.h"
|
|
||||||
#include "xdb_api.h"
|
|
||||||
|
|
||||||
typedef void (* test_func_ptr) ();
|
|
||||||
struct test_func_entry {
|
|
||||||
char *name;
|
|
||||||
test_func_ptr func;
|
|
||||||
};
|
|
||||||
typedef struct test_func_entry test_func_t;
|
|
||||||
|
|
||||||
void test_load_header() {
|
|
||||||
xdb_header_t *header = xdb_load_header_from_file("../../data/ip2region_v4.xdb");
|
|
||||||
if (header == NULL) {
|
|
||||||
printf("failed to load header");
|
|
||||||
} else {
|
|
||||||
printf("header loaded: {\n"
|
|
||||||
" version: %d, \n"
|
|
||||||
" index_policy: %d, \n"
|
|
||||||
" created_at: %u, \n"
|
|
||||||
" start_index_ptr: %d, \n"
|
|
||||||
" end_index_ptr: %d\n"
|
|
||||||
" ip_version: %d\n"
|
|
||||||
" runtime_ptr_bytes: %d\n"
|
|
||||||
" length: %d\n"
|
|
||||||
"}\n",
|
|
||||||
header->version, header->index_policy, header->created_at,
|
|
||||||
header->start_index_ptr, header->end_index_ptr,
|
|
||||||
header->ip_version, header->runtime_ptr_bytes, header->length
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
xdb_free_header(header);
|
|
||||||
}
|
|
||||||
|
|
||||||
void test_load_vector_index() {
|
|
||||||
xdb_vector_index_t *v_index = xdb_load_vector_index_from_file("../../data/ip2region_v4.xdb");
|
|
||||||
if (v_index == NULL) {
|
|
||||||
printf("failed to load vector index from file\n");
|
|
||||||
} else {
|
|
||||||
printf("vector index loaded from file, length=%d\n", v_index->length);
|
|
||||||
}
|
|
||||||
|
|
||||||
xdb_free_vector_index(v_index);
|
|
||||||
}
|
|
||||||
|
|
||||||
void test_load_content() {
|
|
||||||
xdb_content_t *content = xdb_load_content_from_file("../../data/ip2region_v4.xdb");
|
|
||||||
if (content == NULL) {
|
|
||||||
printf("failed to load content from file\n");
|
|
||||||
} else {
|
|
||||||
printf("content loaded from file, length=%d\n", content->length);
|
|
||||||
}
|
|
||||||
|
|
||||||
xdb_free_content(content);
|
|
||||||
}
|
|
||||||
|
|
||||||
void test_parse_ip() {
|
|
||||||
const char *ip_list[] = {
|
|
||||||
"1.0.0.0", "58.251.30.115", "192.168.1.100",
|
|
||||||
"::", "2c0f:fff0::", "2fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff", "240e:982:e617:ffff:ffff:ffff:ffff:ffff",
|
|
||||||
"219.xx.xx.11", "::xx:ffff",
|
|
||||||
NULL
|
|
||||||
};
|
|
||||||
|
|
||||||
int errcode;
|
|
||||||
xdb_version_t *version;
|
|
||||||
bytes_ip_t ip_bytes[16] = {'\0'};
|
|
||||||
string_ip_t ip_string[INET6_ADDRSTRLEN] = {'\0'};
|
|
||||||
|
|
||||||
// init the sock env (for windows)
|
|
||||||
if ((errcode = xdb_init_winsock()) != 0) {
|
|
||||||
printf("failed to init the winsock");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
for (int i = 0;; i++) {
|
|
||||||
if (ip_list[i] == NULL) {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
|
|
||||||
version = xdb_parse_ip(ip_list[i], ip_bytes, sizeof(ip_bytes));
|
|
||||||
if (version == NULL) {
|
|
||||||
printf("failed to parse ip `%s`\n", ip_list[i]);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
xdb_ip_to_string(ip_bytes, version->bytes, ip_string, sizeof(ip_string));
|
|
||||||
printf("ip: %s (version=v%d), toString: %s\n", ip_list[i], version->id, ip_string);
|
|
||||||
}
|
|
||||||
|
|
||||||
// clean up the winsock
|
|
||||||
xdb_clean_winsock();
|
|
||||||
}
|
|
||||||
|
|
||||||
struct ip_pair {
|
|
||||||
const char *sip;
|
|
||||||
const char *eip;
|
|
||||||
};
|
|
||||||
void test_ip_compare() {
|
|
||||||
struct ip_pair ip_pair_list[] = {
|
|
||||||
{"1.0.0.0", "1.0.0.1"},
|
|
||||||
{"192.168.1.101", "192.168.1.90"},
|
|
||||||
{"219.133.111.87", "114.114.114.114"},
|
|
||||||
{"1.0.4.0", "1.0.1.0"},
|
|
||||||
{"1.0.4.0", "1.0.3.255"},
|
|
||||||
{"2000::", "2000:ffff:ffff:ffff:ffff:ffff:ffff:ffff"},
|
|
||||||
{"2001:4:112::", "2001:4:112:ffff:ffff:ffff:ffff:ffff"},
|
|
||||||
{"ffff::", "2001:4:ffff:ffff:ffff:ffff:ffff:ffff"},
|
|
||||||
{NULL, NULL}
|
|
||||||
};
|
|
||||||
|
|
||||||
struct ip_pair *pair_ptr = NULL;
|
|
||||||
bytes_ip_t sip_bytes[16] = {'\0'};
|
|
||||||
bytes_ip_t eip_bytes[16] = {'\0'};
|
|
||||||
xdb_version_t *s_version, *e_version;
|
|
||||||
int errcode;
|
|
||||||
|
|
||||||
// init the sock env (for windows)
|
|
||||||
if ((errcode = xdb_init_winsock()) != 0) {
|
|
||||||
printf("failed to init the winsock");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
for (int i = 0; ;i++) {
|
|
||||||
pair_ptr = &ip_pair_list[i];
|
|
||||||
if (pair_ptr->sip == NULL) {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
|
|
||||||
s_version = xdb_parse_ip(pair_ptr->sip, sip_bytes, sizeof(sip_bytes));
|
|
||||||
if (s_version == NULL) {
|
|
||||||
printf("failed to parse sip `%s`", pair_ptr->sip);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
e_version = xdb_parse_ip(pair_ptr->eip, eip_bytes, sizeof(eip_bytes));
|
|
||||||
if (e_version == NULL) {
|
|
||||||
printf("failed to parse eip `%s`", pair_ptr->eip);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (s_version->id != e_version->id) {
|
|
||||||
printf("sip and eip version not match `%s` != `%s`\n", s_version->name, e_version->name);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
printf(
|
|
||||||
"ip_sub_compare(%s, %s): %d\n",
|
|
||||||
pair_ptr->sip, pair_ptr->eip,
|
|
||||||
xdb_ip_sub_compare(sip_bytes, s_version->bytes, (string_ip_t *) eip_bytes, 0)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
// clean up the winsock
|
|
||||||
xdb_clean_winsock();
|
|
||||||
}
|
|
||||||
|
|
||||||
// please register your function heare
|
|
||||||
static test_func_t _test_function_list[] = {
|
|
||||||
// xdb buffer
|
|
||||||
{"test_load_header", test_load_header},
|
|
||||||
{"test_load_vector_index", test_load_vector_index},
|
|
||||||
{"test_load_content", test_load_content},
|
|
||||||
|
|
||||||
// ip utils
|
|
||||||
{"test_parse_ip", test_parse_ip},
|
|
||||||
{"test_ip_compare", test_ip_compare},
|
|
||||||
|
|
||||||
{NULL, NULL}
|
|
||||||
};
|
|
||||||
|
|
||||||
// valgrind --tool=memcheck --leak-check=full ./a.out
|
|
||||||
int main(int argc, char *argv[]) {
|
|
||||||
int i;
|
|
||||||
char *name;
|
|
||||||
|
|
||||||
// check and call the function
|
|
||||||
if (argc < 2) {
|
|
||||||
printf("please specified the function name to call\n");
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
name = argv[1];
|
|
||||||
test_func_ptr func = NULL;
|
|
||||||
for (i = 0; ; i++) {
|
|
||||||
if (_test_function_list[i].name == NULL) {
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (strcmp(name, _test_function_list[i].name) == 0) {
|
|
||||||
func = _test_function_list[i].func;
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (func == NULL) {
|
|
||||||
printf("can't find test function `%s`\n", name);
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// call the function
|
|
||||||
func();
|
|
||||||
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
@ -0,0 +1,101 @@
|
||||||
|
// Copyright 2022 The Ip2Region Authors. All rights reserved.
|
||||||
|
// Use of this source code is governed by a Apache2.0-style
|
||||||
|
// license that can be found in the LICENSE file.
|
||||||
|
|
||||||
|
// ---
|
||||||
|
// @Author Lion <chenxin619315@gmail.com>
|
||||||
|
// @Date 2022/06/27
|
||||||
|
|
||||||
|
#include "stdio.h"
|
||||||
|
#include "xdb_searcher.h"
|
||||||
|
|
||||||
|
void test_check_ip() {
|
||||||
|
char *ip_list[] = {
|
||||||
|
"1.2.3.4", "192.168.2.3", "120.24.78.129", "255.255.255.0",
|
||||||
|
"256.7.12.9", "12.56.78.320", "32.12.45.192", "222.221.220.219",
|
||||||
|
"192.168.1.101 ", "132.96.12.98a", "x23.12.2.12"
|
||||||
|
};
|
||||||
|
|
||||||
|
int errcode, i;
|
||||||
|
unsigned int ip;
|
||||||
|
char ip_buff[16] = {'\0'};
|
||||||
|
for (i = 0; i < 11; i++) {
|
||||||
|
errcode = xdb_check_ip(ip_list[i], &ip);
|
||||||
|
if (errcode != 0) {
|
||||||
|
printf("invalid ip address `%s`\n", ip_list[i]);
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
xdb_long2ip(ip, ip_buff);
|
||||||
|
printf("long(%-15s)=%-10u, long2ip(%-10u)=%-15s", ip_list[i], ip, ip, ip_buff);
|
||||||
|
if (strcmp(ip_list[i], ip_buff) != 0) {
|
||||||
|
printf(" --[Failed]\n");
|
||||||
|
} else {
|
||||||
|
printf(" --[Ok]\n");
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void test_load_header() {
|
||||||
|
xdb_header_t *header = xdb_load_header_from_file("../../data/ip2region.xdb");
|
||||||
|
if (header == NULL) {
|
||||||
|
printf("failed to load header");
|
||||||
|
} else {
|
||||||
|
printf("header loaded: {\n"
|
||||||
|
" version: %d, \n"
|
||||||
|
" index_policy: %d, \n"
|
||||||
|
" created_at: %u, \n"
|
||||||
|
" start_index_ptr: %d, \n"
|
||||||
|
" end_index_ptr: %d\n"
|
||||||
|
" length: %d\n"
|
||||||
|
"}\n",
|
||||||
|
header->version, header->index_policy, header->created_at,
|
||||||
|
header->start_index_ptr, header->end_index_ptr, header->length
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
xdb_close_header(header);
|
||||||
|
}
|
||||||
|
|
||||||
|
void test_load_vector_index() {
|
||||||
|
xdb_vector_index_t *v_index = xdb_load_vector_index_from_file("../../data/ip2region.xdb");
|
||||||
|
if (v_index == NULL) {
|
||||||
|
printf("failed to load vector index from file\n");
|
||||||
|
} else {
|
||||||
|
printf("vector index loaded from file, length=%d\n", v_index->length);
|
||||||
|
}
|
||||||
|
|
||||||
|
xdb_close_vector_index(v_index);
|
||||||
|
}
|
||||||
|
|
||||||
|
void test_load_content() {
|
||||||
|
xdb_content_t *content = xdb_load_content_from_file("../../data/ip2region.xdb");
|
||||||
|
if (content == NULL) {
|
||||||
|
printf("failed to load content from file\n");
|
||||||
|
} else {
|
||||||
|
printf("content loaded from file, length=%d\n", content->length);
|
||||||
|
}
|
||||||
|
|
||||||
|
xdb_close_content(content);
|
||||||
|
}
|
||||||
|
|
||||||
|
// valgrind --tool=memcheck --leak-check=full ./a.out
|
||||||
|
int main(int argc, char *argv[]) {
|
||||||
|
printf("test check ip ... \n");
|
||||||
|
test_check_ip();
|
||||||
|
printf("|--done\n\n");
|
||||||
|
|
||||||
|
printf("test load header ... \n");
|
||||||
|
test_load_header();
|
||||||
|
printf("|--done\n\n");
|
||||||
|
|
||||||
|
printf("test load vector index ... \n");
|
||||||
|
test_load_vector_index();
|
||||||
|
printf("|--done\n\n");
|
||||||
|
|
||||||
|
printf("test load content ... \n");
|
||||||
|
test_load_content();
|
||||||
|
printf("|--done\n\n");
|
||||||
|
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
@ -1,285 +0,0 @@
|
||||||
// Copyright 2022 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
|
|
||||||
// ---
|
|
||||||
// @Author Lion <chenxin619315@gmail.com>
|
|
||||||
// @Date 2022/06/27
|
|
||||||
|
|
||||||
#ifndef C_IP2REGION_XDB_H
|
|
||||||
#define C_IP2REGION_XDB_H
|
|
||||||
|
|
||||||
// @Note:
|
|
||||||
// this define must be put before any header include
|
|
||||||
// force the LFS for ftell
|
|
||||||
#define _FILE_OFFSET_BITS 64
|
|
||||||
#include <stdio.h>
|
|
||||||
#include <stdlib.h>
|
|
||||||
#include <string.h>
|
|
||||||
|
|
||||||
#if ( defined(WIN32) || defined(_WIN32) || defined(__WINDOWS_) || defined(WINNT) )
|
|
||||||
# define XDB_PUBLIC(type) extern __declspec(dllexport) type
|
|
||||||
# define XDB_PRIVATE(type) static type
|
|
||||||
# define XDB_WINDOWS
|
|
||||||
#include <windows.h>
|
|
||||||
#include <winsock2.h>
|
|
||||||
#include <ws2tcpip.h>
|
|
||||||
#pragma comment(lib, "ws2_32.lib")
|
|
||||||
#elif (defined(linux) || defined(_UNIX) || defined(__APPLE__) || defined(unix) || defined(__unix) || defined(__unix__) || defined(__linux__) || defined(linux) || defined(__linux))
|
|
||||||
# define XDB_PUBLIC(type) extern type
|
|
||||||
# define XDB_PRIVATE(type) static inline type
|
|
||||||
# define XDB_LINUX
|
|
||||||
#include <sys/socket.h>
|
|
||||||
#include <netinet/in.h>
|
|
||||||
#include <arpa/inet.h>
|
|
||||||
#else
|
|
||||||
# define XDB_PUBLIC(type) type
|
|
||||||
# define XDB_PRIVATE(type) static type
|
|
||||||
#endif
|
|
||||||
|
|
||||||
#define xdb_calloc( _blocks, _bytes ) calloc( _blocks, _bytes )
|
|
||||||
#define xdb_malloc( _bytes ) malloc( _bytes )
|
|
||||||
#define xdb_free( _ptr ) free( _ptr )
|
|
||||||
|
|
||||||
// public constants define
|
|
||||||
#define xdb_structure_20 2
|
|
||||||
#define xdb_structure_30 3
|
|
||||||
#define xdb_header_info_length 256
|
|
||||||
#define xdb_vector_index_rows 256
|
|
||||||
#define xdb_vector_index_cols 256
|
|
||||||
#define xdb_vector_index_size 8
|
|
||||||
#define xdb_v4_index_size 14 // 4 + 4 + 2 + 4
|
|
||||||
#define xdb_v6_index_size 38 // 16 + 16 + 2 + 4
|
|
||||||
|
|
||||||
// --- ip version info
|
|
||||||
#define xdb_ipv4_id 4
|
|
||||||
#define xdb_ipv6_id 6
|
|
||||||
#define xdb_ipv4_bytes 4
|
|
||||||
#define xdb_ipv6_bytes 16
|
|
||||||
// cache of vector_index_row × vector_index_rows × vector_index_size
|
|
||||||
#define xdb_vector_index_length 524288
|
|
||||||
|
|
||||||
// --- xdb buffer functions
|
|
||||||
|
|
||||||
// use the following buffer struct to wrap the binary buffer data
|
|
||||||
// since the buffer data could not be operated with the string API.
|
|
||||||
struct xdb_header {
|
|
||||||
unsigned short version;
|
|
||||||
unsigned short index_policy;
|
|
||||||
unsigned int created_at;
|
|
||||||
unsigned int start_index_ptr;
|
|
||||||
unsigned int end_index_ptr;
|
|
||||||
|
|
||||||
// since 3.0+ with IPv6 supporting
|
|
||||||
unsigned short ip_version;
|
|
||||||
unsigned short runtime_ptr_bytes;
|
|
||||||
|
|
||||||
// the original buffer
|
|
||||||
unsigned int length;
|
|
||||||
char buffer[xdb_header_info_length];
|
|
||||||
};
|
|
||||||
typedef struct xdb_header xdb_header_t;
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_header_t *) xdb_load_header(FILE *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_header_t *) xdb_load_header_from_file(const char *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_free_header(void *);
|
|
||||||
|
|
||||||
|
|
||||||
// --- vector index buffer
|
|
||||||
struct xdb_vector_index {
|
|
||||||
unsigned int length;
|
|
||||||
char buffer[xdb_vector_index_length];
|
|
||||||
};
|
|
||||||
typedef struct xdb_vector_index xdb_vector_index_t;
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index(FILE *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index_from_file(const char *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_free_vector_index(void *);
|
|
||||||
|
|
||||||
|
|
||||||
// --- content buffer
|
|
||||||
struct xdb_content {
|
|
||||||
unsigned int length;
|
|
||||||
char *buffer;
|
|
||||||
};
|
|
||||||
typedef struct xdb_content xdb_content_t;
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_content_t *) xdb_load_content(FILE *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_content_t *) xdb_load_content_from_file(const char *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_free_content(void *);
|
|
||||||
|
|
||||||
// --- xdb verify
|
|
||||||
|
|
||||||
// Verify if the current Searcher could be used to search the specified xdb file.
|
|
||||||
// Why do we need this check ?
|
|
||||||
// The future features of the xdb impl may cause the current searcher not able to work properly.
|
|
||||||
//
|
|
||||||
// @Note: You Just need to check this ONCE when the service starts
|
|
||||||
// Or use another process (eg, A command) to check once Just to confirm the suitability.
|
|
||||||
XDB_PUBLIC(int) xdb_verify(FILE *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_verify_from_header(FILE *handle, xdb_header_t *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_verify_from_file(const char *);
|
|
||||||
|
|
||||||
// --- End xdb buffer
|
|
||||||
|
|
||||||
|
|
||||||
// types type define
|
|
||||||
typedef char string_ip_t;
|
|
||||||
typedef unsigned char bytes_ip_t;
|
|
||||||
|
|
||||||
// --- ip version
|
|
||||||
#define XDB_IPv4 (xdb_version_v4())
|
|
||||||
#define XDB_IPv6 (xdb_version_v6())
|
|
||||||
typedef int (* ip_compare_fn_t) (const bytes_ip_t *, int, const char *, int);
|
|
||||||
struct xdb_ip_version_entry {
|
|
||||||
int id; // version id
|
|
||||||
char *name; // version name
|
|
||||||
int bytes; // ip bytes number
|
|
||||||
int segment_index_size; // segment index size in bytes
|
|
||||||
|
|
||||||
// function to compare two ips
|
|
||||||
ip_compare_fn_t ip_compare;
|
|
||||||
};
|
|
||||||
typedef struct xdb_ip_version_entry xdb_version_t;
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_version_v4();
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_version_v6();
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_version_is_v4(const xdb_version_t *);
|
|
||||||
XDB_PUBLIC(int) xdb_version_is_v6(const xdb_version_t *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_version_from_name(char *);
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_version_from_header(xdb_header_t *);
|
|
||||||
|
|
||||||
// --- END ip version
|
|
||||||
|
|
||||||
// --- xdb util functions
|
|
||||||
|
|
||||||
// to compatiable with the windows
|
|
||||||
// returns: 0 for ok and -1 for failed
|
|
||||||
XDB_PUBLIC(int) xdb_init_winsock();
|
|
||||||
XDB_PUBLIC(void) xdb_clean_winsock();
|
|
||||||
|
|
||||||
// get the current time in microseconds
|
|
||||||
XDB_PUBLIC(long) xdb_now();
|
|
||||||
|
|
||||||
// get unsigned long (4bytes) from a specified buffer start from the specified offset with little-endian
|
|
||||||
XDB_PUBLIC(unsigned int) xdb_le_get_uint32(const char *, int);
|
|
||||||
|
|
||||||
// get unsigned short (2bytes) from a specified buffer start from the specified offset with little-endian
|
|
||||||
XDB_PUBLIC(int) xdb_le_get_uint16(const char *, int);
|
|
||||||
|
|
||||||
|
|
||||||
// parse the specified IP address to byte array.
|
|
||||||
// returns: xdb_version_t for valid ipv4 / ipv6, or NULL for failed
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_parse_ip(const string_ip_t *, bytes_ip_t *, size_t);
|
|
||||||
|
|
||||||
// parse the specified IPv4 address to byte array
|
|
||||||
// returns: xdb_version_t for valid ipv4, or NULL for failed
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_parse_v4_ip(const string_ip_t *, bytes_ip_t *, size_t);
|
|
||||||
|
|
||||||
// parse the specified IPv6 address to byte array
|
|
||||||
// returns: xdb_version_t for valid ipv6, or NULL for failed
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_parse_v6_ip(const string_ip_t *, bytes_ip_t *, size_t);
|
|
||||||
|
|
||||||
// convert a specified ip bytes to humen-readable string.
|
|
||||||
// returns: 0 for success or -1 for failed.
|
|
||||||
XDB_PUBLIC(int) xdb_ip_to_string(const bytes_ip_t *, int, char *, size_t);
|
|
||||||
|
|
||||||
// ipv4 bytes to string
|
|
||||||
XDB_PUBLIC(int) xdb_v4_ip_to_string(const bytes_ip_t *, char *, size_t);
|
|
||||||
|
|
||||||
// ipv6 bytes to string
|
|
||||||
XDB_PUBLIC(int) xdb_v6_ip_to_string(const bytes_ip_t *, char *, size_t);
|
|
||||||
|
|
||||||
// compare the specified ip bytes with another ip bytes in the specified buff from offset.
|
|
||||||
// ip args must be the return value from #xdb_parse_ip.
|
|
||||||
// returns: -1 if ip1 < ip2, 1 if ip1 > ip2 or 0
|
|
||||||
XDB_PUBLIC(int) xdb_ip_sub_compare(const bytes_ip_t *, int, const char *, int);
|
|
||||||
|
|
||||||
// large file seek and tell
|
|
||||||
XDB_PUBLIC(int) xdb_fseek(FILE *, long long, int);
|
|
||||||
|
|
||||||
XDB_PUBLIC(long long) xdb_ftell(FILE *);
|
|
||||||
|
|
||||||
// --- END xdb utils
|
|
||||||
|
|
||||||
|
|
||||||
// --- xdb searcher api
|
|
||||||
|
|
||||||
// xdb region info structure
|
|
||||||
#define xdb_region_buffer_wrapper 1
|
|
||||||
#define xdb_region_buffer_auto 2
|
|
||||||
struct xdb_region_buffer_entry {
|
|
||||||
int type; // buffer type
|
|
||||||
char *value; // region value
|
|
||||||
size_t length; // buffer length
|
|
||||||
};
|
|
||||||
typedef struct xdb_region_buffer_entry xdb_region_buffer_t;
|
|
||||||
|
|
||||||
// wrapper the region from a local stack buffer.
|
|
||||||
// returns: 0 for succeed or failed
|
|
||||||
XDB_PUBLIC(int) xdb_region_buffer_init(xdb_region_buffer_t *, char *, size_t);
|
|
||||||
|
|
||||||
// do the buffer alloc.
|
|
||||||
// returns: 0 for ok or failed
|
|
||||||
XDB_PUBLIC(int) xdb_region_buffer_alloc(xdb_region_buffer_t *, int);
|
|
||||||
|
|
||||||
// empty alloc - empty string
|
|
||||||
// returns: 0 - always
|
|
||||||
XDB_PUBLIC(int) xdb_region_buffer_empty(xdb_region_buffer_t *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_region_buffer_free(xdb_region_buffer_t *);
|
|
||||||
|
|
||||||
// xdb searcher structure
|
|
||||||
struct xdb_searcher_entry {
|
|
||||||
// ip version
|
|
||||||
xdb_version_t *version;
|
|
||||||
|
|
||||||
// xdb file handle
|
|
||||||
FILE *handle;
|
|
||||||
|
|
||||||
// header info
|
|
||||||
const char *header;
|
|
||||||
int io_count;
|
|
||||||
|
|
||||||
// vector index buffer cache.
|
|
||||||
// preload the vector index will reduce the number of IO operations
|
|
||||||
// thus speedup the search process.
|
|
||||||
const xdb_vector_index_t *v_index;
|
|
||||||
|
|
||||||
// content buffer.
|
|
||||||
// cache the whole xdb content.
|
|
||||||
const xdb_content_t *content;
|
|
||||||
};
|
|
||||||
typedef struct xdb_searcher_entry xdb_searcher_t;
|
|
||||||
|
|
||||||
// xdb searcher new api define
|
|
||||||
XDB_PUBLIC(int) xdb_new_with_file_only(xdb_version_t *, xdb_searcher_t *, const char *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_new_with_vector_index(xdb_version_t *, xdb_searcher_t *, const char *, const xdb_vector_index_t *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_new_with_buffer(xdb_version_t *, xdb_searcher_t *, const xdb_content_t *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_close(void *);
|
|
||||||
|
|
||||||
// xdb searcher search api define
|
|
||||||
XDB_PUBLIC(int) xdb_search_by_string(xdb_searcher_t *, const string_ip_t *, xdb_region_buffer_t *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_search(xdb_searcher_t *, const bytes_ip_t *, int, xdb_region_buffer_t *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_get_version(xdb_searcher_t *);
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_get_io_count(xdb_searcher_t *);
|
|
||||||
|
|
||||||
// --- END xdb searcher api
|
|
||||||
|
|
||||||
#endif // C_IP2REGION_XDB_H
|
|
||||||
|
|
@ -6,104 +6,42 @@
|
||||||
// @Author Lion <chenxin619315@gmail.com>
|
// @Author Lion <chenxin619315@gmail.com>
|
||||||
// @Date 2022/06/27
|
// @Date 2022/06/27
|
||||||
|
|
||||||
#include "xdb_api.h"
|
#include "xdb_searcher.h"
|
||||||
|
|
||||||
// --- region buffer
|
// for Linux
|
||||||
XDB_PUBLIC(int) xdb_region_buffer_init(xdb_region_buffer_t *region, char *buffer, size_t length) {
|
#ifdef XDB_LINUX
|
||||||
if (buffer == NULL) {
|
#include "sys/time.h"
|
||||||
region->type = xdb_region_buffer_auto;
|
#endif
|
||||||
region->length = 0;
|
|
||||||
} else if (length <= 0) {
|
|
||||||
return 1;
|
|
||||||
} else {
|
|
||||||
region->type = xdb_region_buffer_wrapper;
|
|
||||||
region->length = length;
|
|
||||||
memset(buffer, 0x00, length); // zero-fill the buffer
|
|
||||||
}
|
|
||||||
|
|
||||||
region->value = buffer;
|
// @Note: since 2023/10/13 to compatible with the windows system
|
||||||
return 0;
|
#ifdef XDB_WINDOWS
|
||||||
|
#include <windows.h>
|
||||||
|
XDB_PRIVATE(int) gettimeofday(struct timeval* tp, void* tzp) {
|
||||||
|
time_t clock;
|
||||||
|
struct tm tm;
|
||||||
|
SYSTEMTIME wtm;
|
||||||
|
GetLocalTime(&wtm);
|
||||||
|
tm.tm_year = wtm.wYear - 1900;
|
||||||
|
tm.tm_mon = wtm.wMonth - 1;
|
||||||
|
tm.tm_mday = wtm.wDay;
|
||||||
|
tm.tm_hour = wtm.wHour;
|
||||||
|
tm.tm_min = wtm.wMinute;
|
||||||
|
tm.tm_sec = wtm.wSecond;
|
||||||
|
tm.tm_isdst = -1;
|
||||||
|
clock = mktime(&tm);
|
||||||
|
tp->tv_sec = clock;
|
||||||
|
tp->tv_usec = wtm.wMilliseconds * 1000;
|
||||||
|
return (0);
|
||||||
}
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_region_buffer_alloc(xdb_region_buffer_t *region, int length) {
|
|
||||||
if (length <= 0) {
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// no allocation supports for the buffer wapper
|
|
||||||
if (region->type == xdb_region_buffer_wrapper) {
|
|
||||||
if (length >= region->length) {
|
|
||||||
return 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
region->value[length] = '\0';
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ensure that the value were freed
|
|
||||||
// by calling #xdb_region_buffer_free
|
|
||||||
if (region->value != NULL) {
|
|
||||||
return 3;
|
|
||||||
}
|
|
||||||
|
|
||||||
char *ptr = (char *) xdb_malloc(length + 1);
|
|
||||||
if (ptr == NULL) {
|
|
||||||
return 4;
|
|
||||||
}
|
|
||||||
|
|
||||||
ptr[length] = '\0'; // NULL-end
|
|
||||||
region->value = ptr;
|
|
||||||
region->length = length;
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
// fixed internal empty string ptr
|
|
||||||
static char * _empty_region_string = "\0";
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_region_buffer_empty(xdb_region_buffer_t *region) {
|
|
||||||
// no allocation supports for the buffer wapper
|
|
||||||
if (region->type == xdb_region_buffer_wrapper) {
|
|
||||||
region->value[0] = '\0';
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ensure that the value were freed
|
|
||||||
// by calling #xdb_region_buffer_free
|
|
||||||
if (region->value != NULL) {
|
|
||||||
return 3;
|
|
||||||
}
|
|
||||||
|
|
||||||
region->value = _empty_region_string;
|
|
||||||
region->length = 0;
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_region_buffer_free(xdb_region_buffer_t *region) {
|
|
||||||
if (region->type == xdb_region_buffer_auto) {
|
|
||||||
// empty string interception
|
|
||||||
if (region->length == 0
|
|
||||||
|| region->value == _empty_region_string) {
|
|
||||||
// do nothing for empty string
|
|
||||||
} else {
|
|
||||||
xdb_free(region->value);
|
|
||||||
}
|
|
||||||
|
|
||||||
// reset the value
|
|
||||||
region->value = NULL;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- END region buffer
|
|
||||||
|
|
||||||
// internal function prototype define
|
// internal function prototype define
|
||||||
XDB_PRIVATE(int) read(xdb_searcher_t *, long offset, char *, size_t length);
|
XDB_PRIVATE(int) read(xdb_searcher_t *, long offset, char *, size_t length);
|
||||||
|
|
||||||
XDB_PRIVATE(int) xdb_new_base(xdb_version_t *version, xdb_searcher_t *xdb, const char *db_path, const xdb_vector_index_t *v_index, const xdb_content_t *c_buffer) {
|
XDB_PRIVATE(int) xdb_new_base(xdb_searcher_t *xdb, const char *db_path, const xdb_vector_index_t *v_index, const xdb_content_t *c_buffer) {
|
||||||
memset(xdb, 0x00, sizeof(xdb_searcher_t));
|
memset(xdb, 0x00, sizeof(xdb_searcher_t));
|
||||||
|
|
||||||
// set the version
|
|
||||||
xdb->version = version;
|
|
||||||
|
|
||||||
// check the content buffer first
|
// check the content buffer first
|
||||||
if (c_buffer != NULL) {
|
if (c_buffer != NULL) {
|
||||||
xdb->v_index = NULL;
|
xdb->v_index = NULL;
|
||||||
|
|
@ -124,16 +62,16 @@ XDB_PRIVATE(int) xdb_new_base(xdb_version_t *version, xdb_searcher_t *xdb, const
|
||||||
}
|
}
|
||||||
|
|
||||||
// xdb searcher new api define
|
// xdb searcher new api define
|
||||||
XDB_PUBLIC(int) xdb_new_with_file_only(xdb_version_t *version, xdb_searcher_t *xdb, const char *db_path) {
|
XDB_PUBLIC(int) xdb_new_with_file_only(xdb_searcher_t *xdb, const char *db_path) {
|
||||||
return xdb_new_base(version, xdb, db_path, NULL, NULL);
|
return xdb_new_base(xdb, db_path, NULL, NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_new_with_vector_index(xdb_version_t *version, xdb_searcher_t *xdb, const char *db_path, const xdb_vector_index_t *v_index) {
|
XDB_PUBLIC(int) xdb_new_with_vector_index(xdb_searcher_t *xdb, const char *db_path, const xdb_vector_index_t *v_index) {
|
||||||
return xdb_new_base(version, xdb, db_path, v_index, NULL);
|
return xdb_new_base(xdb, db_path, v_index, NULL);
|
||||||
}
|
}
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_new_with_buffer(xdb_version_t *version, xdb_searcher_t *xdb, const xdb_content_t *c_buffer) {
|
XDB_PUBLIC(int) xdb_new_with_buffer(xdb_searcher_t *xdb, const xdb_content_t *c_buffer) {
|
||||||
return xdb_new_base(version, xdb, NULL, NULL, c_buffer);
|
return xdb_new_base(xdb, NULL, NULL, c_buffer);
|
||||||
}
|
}
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_close(void *ptr) {
|
XDB_PUBLIC(void) xdb_close(void *ptr) {
|
||||||
|
|
@ -146,109 +84,93 @@ XDB_PUBLIC(void) xdb_close(void *ptr) {
|
||||||
|
|
||||||
// --- xdb searcher search api define
|
// --- xdb searcher search api define
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_search_by_string(xdb_searcher_t *xdb, const string_ip_t *ip_string, xdb_region_buffer_t *region) {
|
XDB_PUBLIC(int) xdb_search_by_string(xdb_searcher_t *xdb, const char *str_ip, char *region_buffer, size_t length) {
|
||||||
bytes_ip_t ip_bytes[16] = {'\0'};
|
unsigned int ip = 0;
|
||||||
xdb_version_t *version = xdb_parse_ip(ip_string, ip_bytes, sizeof(ip_bytes));
|
int errcode = xdb_check_ip(str_ip, &ip);
|
||||||
if (version == NULL) {
|
if (errcode != 0) {
|
||||||
return 10;
|
return 10 + errcode;
|
||||||
} else {
|
} else {
|
||||||
return xdb_search(xdb, ip_bytes, version->bytes, region);
|
return xdb_search(xdb, ip, region_buffer, length);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_search(xdb_searcher_t *xdb, const bytes_ip_t *ip_bytes, int ip_len, xdb_region_buffer_t *region) {
|
XDB_PUBLIC(int) xdb_search(xdb_searcher_t *xdb, unsigned int ip, char *region_buffer, size_t length) {
|
||||||
int il0, il1, idx, err, bytes, d_bytes;
|
int il0, il1, idx, err, l, h, m, data_len;
|
||||||
register int seg_index_size, l, h, m, p;
|
unsigned int s_ptr, e_ptr, p, sip, eip, data_ptr;
|
||||||
unsigned int s_ptr, e_ptr, data_ptr, data_len;
|
char vector_buffer[xdb_vector_index_size], segment_buffer[xdb_segment_index_size];
|
||||||
char vector_buffer[xdb_vector_index_size];
|
|
||||||
char segment_buffer[xdb_v6_index_size];
|
|
||||||
|
|
||||||
// ip version check
|
// reset the io counter
|
||||||
if (ip_len != xdb->version->bytes) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// some resets
|
|
||||||
err = 0;
|
|
||||||
data_len = 0;
|
|
||||||
bytes = xdb->version->bytes;
|
|
||||||
d_bytes = xdb->version->bytes << 1;
|
|
||||||
xdb->io_count = 0;
|
xdb->io_count = 0;
|
||||||
|
|
||||||
// locate the segment index block based on the vector index
|
// locate the segment index block based on the vector index
|
||||||
il0 = (int) (ip_bytes[0]);
|
il0 = ((int) (ip >> 24)) & 0xFF;
|
||||||
il1 = (int) (ip_bytes[1]);
|
il1 = ((int) (ip >> 16)) & 0xFF;
|
||||||
idx = il0 * xdb_vector_index_cols * xdb_vector_index_size + il1 * xdb_vector_index_size;
|
idx = il0 * xdb_vector_index_cols * xdb_vector_index_size + il1 * xdb_vector_index_size;
|
||||||
if (xdb->v_index != NULL) {
|
if (xdb->v_index != NULL) {
|
||||||
s_ptr = xdb_le_get_uint32(xdb->v_index->buffer, idx);
|
s_ptr = xdb_get_uint(xdb->v_index->buffer, idx);
|
||||||
e_ptr = xdb_le_get_uint32(xdb->v_index->buffer, idx + 4);
|
e_ptr = xdb_get_uint(xdb->v_index->buffer, idx + 4);
|
||||||
} else if (xdb->content != NULL) {
|
} else if (xdb->content != NULL) {
|
||||||
s_ptr = xdb_le_get_uint32(xdb->content->buffer, xdb_header_info_length + idx);
|
s_ptr = xdb_get_uint(xdb->content->buffer, xdb_header_info_length + idx);
|
||||||
e_ptr = xdb_le_get_uint32(xdb->content->buffer, xdb_header_info_length + idx + 4);
|
e_ptr = xdb_get_uint(xdb->content->buffer, xdb_header_info_length + idx + 4);
|
||||||
} else {
|
} else {
|
||||||
err = read(xdb, xdb_header_info_length + idx, vector_buffer, sizeof(vector_buffer));
|
err = read(xdb, xdb_header_info_length + idx, vector_buffer, sizeof(vector_buffer));
|
||||||
if (err != 0) {
|
if (err != 0) {
|
||||||
return 10 + err;
|
return 10 + err;
|
||||||
}
|
}
|
||||||
|
|
||||||
s_ptr = xdb_le_get_uint32(vector_buffer, 0);
|
s_ptr = xdb_get_uint(vector_buffer, 0);
|
||||||
e_ptr = xdb_le_get_uint32(vector_buffer, 4);
|
e_ptr = xdb_get_uint(vector_buffer, 4);
|
||||||
}
|
}
|
||||||
|
|
||||||
// printf("s_ptr=%u, e_ptr=%u\n", s_ptr, e_ptr);
|
// printf("s_ptr=%u, e_ptr=%u\n", s_ptr, e_ptr);
|
||||||
// @Note: ptr validate, zero ptr means source data missing
|
|
||||||
// so we could just stop here and return an empty string.
|
|
||||||
if (s_ptr == 0 || e_ptr == 0) {
|
|
||||||
xdb_region_buffer_empty(region);
|
|
||||||
return err;
|
|
||||||
}
|
|
||||||
|
|
||||||
// binary search to get the final region info
|
// binary search to get the final region info
|
||||||
// segment_buffer = xdb_malloc(seg_index_size);
|
|
||||||
seg_index_size = xdb->version->segment_index_size;
|
|
||||||
data_len = 0, data_ptr = 0;
|
data_len = 0, data_ptr = 0;
|
||||||
l = 0, h = ((int) (e_ptr - s_ptr)) / seg_index_size;
|
l = 0, h = ((int) (e_ptr - s_ptr)) / xdb_segment_index_size;
|
||||||
while (l <= h) {
|
while (l <= h) {
|
||||||
m = (l + h) >> 1;
|
m = (l + h) >> 1;
|
||||||
p = s_ptr + m * seg_index_size;
|
p = s_ptr + m * xdb_segment_index_size;
|
||||||
|
|
||||||
// read the segment index item
|
// read the segment index item
|
||||||
err = read(xdb, p, segment_buffer, seg_index_size);
|
err = read(xdb, p, segment_buffer, sizeof(segment_buffer));
|
||||||
if (err != 0) {
|
if (err != 0) {
|
||||||
return 20 + err;
|
return 20 + err;
|
||||||
}
|
}
|
||||||
|
|
||||||
// decode the data fields as needed
|
// decode the data fields as needed
|
||||||
if (xdb->version->ip_compare(ip_bytes, bytes, segment_buffer, 0) < 0) {
|
sip = xdb_get_uint(segment_buffer, 0);
|
||||||
|
if (ip < sip) {
|
||||||
h = m - 1;
|
h = m - 1;
|
||||||
} else if (xdb->version->ip_compare(ip_bytes, bytes, segment_buffer, bytes) > 0) {
|
|
||||||
l = m + 1;
|
|
||||||
} else {
|
} else {
|
||||||
data_len = xdb_le_get_uint16(segment_buffer, d_bytes);
|
eip = xdb_get_uint(segment_buffer, 4);
|
||||||
data_ptr = xdb_le_get_uint32(segment_buffer, d_bytes + 2);
|
if (ip > eip) {
|
||||||
break;
|
l = m + 1;
|
||||||
|
} else {
|
||||||
|
data_len = xdb_get_ushort(segment_buffer, 8);
|
||||||
|
data_ptr = xdb_get_uint(segment_buffer, 10);
|
||||||
|
break;
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// printf("data_len=%u, data_ptr=%u\n", data_len, data_ptr);
|
// printf("data_len=%u, data_ptr=%u\n", data_len, data_ptr);
|
||||||
if (data_len == 0) {
|
if (data_len == 0) {
|
||||||
// return 100;
|
region_buffer[0] = '\0';
|
||||||
xdb_region_buffer_empty(region);
|
return 0;
|
||||||
return err;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// buffer alloc checking
|
// buffer length checking
|
||||||
err = xdb_region_buffer_alloc(region, data_len);
|
if (data_len >= (int) length) {
|
||||||
if (err != 0) {
|
return 1;
|
||||||
return 100 + err;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
err = read(xdb, data_ptr, region->value, data_len);
|
err = read(xdb, data_ptr, region_buffer, data_len);
|
||||||
if (err != 0) {
|
if (err != 0) {
|
||||||
return 30 + err;
|
return 30 + err;
|
||||||
}
|
}
|
||||||
|
|
||||||
return err;
|
// auto append a NULL-end
|
||||||
|
region_buffer[data_len] = '\0';
|
||||||
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
XDB_PRIVATE(int) read(xdb_searcher_t *xdb, long offset, char *buffer, size_t length) {
|
XDB_PRIVATE(int) read(xdb_searcher_t *xdb, long offset, char *buffer, size_t length) {
|
||||||
|
|
@ -271,10 +193,237 @@ XDB_PRIVATE(int) read(xdb_searcher_t *xdb, long offset, char *buffer, size_t len
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_get_version(xdb_searcher_t *xdb) {
|
|
||||||
return xdb->version;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_get_io_count(xdb_searcher_t *xdb) {
|
XDB_PUBLIC(int) xdb_get_io_count(xdb_searcher_t *xdb) {
|
||||||
return xdb->io_count;
|
return xdb->io_count;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
// --- buffer load util functions
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_header_t *) xdb_load_header(FILE *handle) {
|
||||||
|
xdb_header_t *header;
|
||||||
|
unsigned int size = xdb_header_info_length;
|
||||||
|
|
||||||
|
// entry alloc
|
||||||
|
header = (xdb_header_t *) xdb_malloc(sizeof(xdb_header_t));
|
||||||
|
if (header == NULL) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (fseek(handle, 0, SEEK_SET) == -1) {
|
||||||
|
xdb_free(header);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (fread(header->buffer, 1,size, handle) != size) {
|
||||||
|
xdb_free(header);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
// fill the fields
|
||||||
|
header->length = size;
|
||||||
|
header->version = (unsigned short) xdb_get_ushort(header->buffer, 0);
|
||||||
|
header->index_policy = (unsigned short) xdb_get_ushort(header->buffer, 2);
|
||||||
|
header->created_at = xdb_get_uint(header->buffer, 4);
|
||||||
|
header->start_index_ptr = xdb_get_uint(header->buffer, 8);
|
||||||
|
header->end_index_ptr = xdb_get_uint(header->buffer,12);
|
||||||
|
|
||||||
|
return header;
|
||||||
|
}
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_header_t *) xdb_load_header_from_file(const char *db_path) {
|
||||||
|
xdb_header_t *header;
|
||||||
|
FILE *handle = fopen(db_path, "rb");
|
||||||
|
if (handle == NULL) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
header = xdb_load_header(handle);
|
||||||
|
fclose(handle);
|
||||||
|
return header;
|
||||||
|
}
|
||||||
|
|
||||||
|
XDB_PUBLIC(void) xdb_close_header(void *ptr) {
|
||||||
|
xdb_header_t *header = (xdb_header_t *) ptr;
|
||||||
|
if (header->length > 0) {
|
||||||
|
header->length = 0;
|
||||||
|
xdb_free(header);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- vector index
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index(FILE *handle) {
|
||||||
|
xdb_vector_index_t *v_index;
|
||||||
|
unsigned int size = xdb_vector_index_length;
|
||||||
|
|
||||||
|
// seek to the vector index offset
|
||||||
|
if (fseek(handle, xdb_header_info_length, SEEK_SET) == -1) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
// do the buffer read
|
||||||
|
v_index = (xdb_vector_index_t *) xdb_malloc(sizeof(xdb_vector_index_t));
|
||||||
|
if (v_index == NULL) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
v_index->length = size;
|
||||||
|
if (fread(v_index->buffer, 1, size, handle) != size) {
|
||||||
|
xdb_free(v_index);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
return v_index;
|
||||||
|
}
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index_from_file(const char *db_path) {
|
||||||
|
xdb_vector_index_t *v_index;
|
||||||
|
FILE *handle = fopen(db_path, "rb");
|
||||||
|
if (handle == NULL) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
v_index = xdb_load_vector_index(handle);
|
||||||
|
fclose(handle);
|
||||||
|
return v_index;
|
||||||
|
}
|
||||||
|
|
||||||
|
XDB_PUBLIC(void) xdb_close_vector_index(void *ptr) {
|
||||||
|
xdb_vector_index_t *v_index = (xdb_vector_index_t *) ptr;
|
||||||
|
if (v_index->length > 0) {
|
||||||
|
v_index->length = 0;
|
||||||
|
xdb_free(v_index);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- content buffer
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_content_t *) xdb_load_content(FILE *handle) {
|
||||||
|
unsigned int size;
|
||||||
|
xdb_content_t *content;
|
||||||
|
|
||||||
|
// determine the file size
|
||||||
|
if (fseek(handle, 0, SEEK_END) == -1) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
size = (unsigned int) ftell(handle);
|
||||||
|
if (fseek(handle, 0, SEEK_SET) == -1) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
// do the file read
|
||||||
|
content = (xdb_content_t *) xdb_malloc(sizeof(xdb_content_t));
|
||||||
|
if (content == NULL) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
// do the buffer alloc
|
||||||
|
content->buffer = (char *) xdb_malloc(size);
|
||||||
|
if (content->buffer == NULL) {
|
||||||
|
xdb_free(content);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
// read the content into the buffer
|
||||||
|
content->length = size;
|
||||||
|
if (fread(content->buffer, 1, size, handle) != size) {
|
||||||
|
xdb_free(content);
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
return content;
|
||||||
|
}
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_content_t *) xdb_load_content_from_file(const char *db_path) {
|
||||||
|
xdb_content_t *content;
|
||||||
|
FILE *handle = fopen(db_path, "rb");
|
||||||
|
if (handle == NULL) {
|
||||||
|
return NULL;
|
||||||
|
}
|
||||||
|
|
||||||
|
content = xdb_load_content(handle);
|
||||||
|
fclose(handle);
|
||||||
|
return content;
|
||||||
|
}
|
||||||
|
|
||||||
|
XDB_PUBLIC(void) xdb_close_content(void *ptr) {
|
||||||
|
xdb_content_t *content = (xdb_content_t *) ptr;
|
||||||
|
if (content->length > 0) {
|
||||||
|
content->length = 0;
|
||||||
|
xdb_free(content->buffer);
|
||||||
|
content->buffer = NULL;
|
||||||
|
xdb_free(content);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- End
|
||||||
|
|
||||||
|
// get unsigned long (4bytes) from a specified buffer start from the specified offset
|
||||||
|
XDB_PUBLIC(unsigned int) xdb_get_uint(const char *buffer, int offset) {
|
||||||
|
return (
|
||||||
|
((buffer[offset ]) & 0x000000FF) |
|
||||||
|
((buffer[offset+1] << 8) & 0x0000FF00) |
|
||||||
|
((buffer[offset+2] << 16) & 0x00FF0000) |
|
||||||
|
((buffer[offset+3] << 24) & 0xFF000000)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// get unsigned short (2bytes) from a specified buffer start from the specified offset
|
||||||
|
XDB_PUBLIC(int) xdb_get_ushort(const char *buffer, int offset) {
|
||||||
|
return (
|
||||||
|
((buffer[offset ]) & 0x000000FF) |
|
||||||
|
((buffer[offset+1] << 8) & 0x0000FF00)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
// string ip to unsigned int
|
||||||
|
static int shiftIndex[4] = {24, 16, 8, 0};
|
||||||
|
XDB_PUBLIC(int) xdb_check_ip(const char *src_ip, unsigned int *dst_ip) {
|
||||||
|
char c;
|
||||||
|
int i, n, ip = 0;
|
||||||
|
const char *ptr = src_ip;
|
||||||
|
for (i = 0; i < 4; i++) {
|
||||||
|
n = 0;
|
||||||
|
while (1) {
|
||||||
|
c = *ptr;
|
||||||
|
ptr++;
|
||||||
|
if (c >= '0' && c <= '9') {
|
||||||
|
n *= 10;
|
||||||
|
n += c - '0';
|
||||||
|
} else if ((i < 3 && c == '.') || i == 3) {
|
||||||
|
// stopping at the '.' but ignore the tailing chars
|
||||||
|
// after the 3rd one (auto clean the tailing none-integer ?).
|
||||||
|
break;
|
||||||
|
} else {
|
||||||
|
return 1;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (n > 0xFF) {
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
ip |= (n << shiftIndex[i]);
|
||||||
|
}
|
||||||
|
|
||||||
|
*dst_ip = ip;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
// unsigned int ip to string ip
|
||||||
|
XDB_PUBLIC(void) xdb_long2ip(unsigned int ip, char *buffer) {
|
||||||
|
sprintf(buffer, "%d.%d.%d.%d", (ip >> 24) & 0xFF, (ip >> 16) & 0xFF, (ip >> 8) & 0xFF, ip & 0xFF);
|
||||||
|
}
|
||||||
|
|
||||||
|
// get the middle ip of a and b
|
||||||
|
XDB_PUBLIC(unsigned int) xdb_mip(unsigned long a, unsigned long b) {
|
||||||
|
return (unsigned int) ((a + b) >> 1);
|
||||||
|
}
|
||||||
|
|
||||||
|
XDB_PUBLIC(long) xdb_now() {
|
||||||
|
struct timeval c_time;
|
||||||
|
gettimeofday(&c_time, NULL);
|
||||||
|
return c_time.tv_sec * (int)1e6 + c_time.tv_usec;
|
||||||
|
}
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,149 @@
|
||||||
|
// Copyright 2022 The Ip2Region Authors. All rights reserved.
|
||||||
|
// Use of this source code is governed by a Apache2.0-style
|
||||||
|
// license that can be found in the LICENSE file.
|
||||||
|
|
||||||
|
// ---
|
||||||
|
// @Author Lion <chenxin619315@gmail.com>
|
||||||
|
// @Date 2022/06/27
|
||||||
|
|
||||||
|
#ifndef C_XDB_SEARCHER_H
|
||||||
|
#define C_XDB_SEARCHER_H
|
||||||
|
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <stdlib.h>
|
||||||
|
#include <string.h>
|
||||||
|
|
||||||
|
#if ( defined(WIN32) || defined(_WIN32) || defined(__WINDOWS_) || defined(WINNT) )
|
||||||
|
# define XDB_PUBLIC(type) extern __declspec(dllexport) type
|
||||||
|
# define XDB_PRIVATE(type) static type
|
||||||
|
# define XDB_WINDOWS
|
||||||
|
#elif ( defined(linux) || defined(_UNIX) || defined(__APPLE__) )
|
||||||
|
# define XDB_PUBLIC(type) extern type
|
||||||
|
# define XDB_PRIVATE(type) static inline type
|
||||||
|
# define XDB_LINUX
|
||||||
|
#endif
|
||||||
|
|
||||||
|
#define xdb_calloc( _blocks, _bytes ) calloc( _blocks, _bytes )
|
||||||
|
#define xdb_malloc( _bytes ) malloc( _bytes )
|
||||||
|
#define xdb_free( _ptr ) free( _ptr )
|
||||||
|
|
||||||
|
// public constants define
|
||||||
|
#define xdb_header_info_length 256
|
||||||
|
#define xdb_vector_index_rows 256
|
||||||
|
#define xdb_vector_index_cols 256
|
||||||
|
#define xdb_vector_index_size 8
|
||||||
|
#define xdb_segment_index_size 14
|
||||||
|
|
||||||
|
// cache of vector_index_row × vector_index_rows × vector_index_size
|
||||||
|
#define xdb_vector_index_length 524288
|
||||||
|
|
||||||
|
|
||||||
|
// --- buffer load util functions
|
||||||
|
|
||||||
|
// use the following buffer struct to wrap the binary buffer data
|
||||||
|
// since the buffer data could not be operated with the string API.
|
||||||
|
struct xdb_header {
|
||||||
|
unsigned short version;
|
||||||
|
unsigned short index_policy;
|
||||||
|
unsigned int created_at;
|
||||||
|
unsigned int start_index_ptr;
|
||||||
|
unsigned int end_index_ptr;
|
||||||
|
|
||||||
|
// the original buffer
|
||||||
|
unsigned int length;
|
||||||
|
char buffer[xdb_header_info_length];
|
||||||
|
};
|
||||||
|
typedef struct xdb_header xdb_header_t;
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_header_t *) xdb_load_header(FILE *);
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_header_t *) xdb_load_header_from_file(const char *);
|
||||||
|
|
||||||
|
XDB_PUBLIC(void) xdb_close_header(void *);
|
||||||
|
|
||||||
|
|
||||||
|
// --- vector index buffer
|
||||||
|
struct xdb_vector_index {
|
||||||
|
unsigned int length;
|
||||||
|
char buffer[xdb_vector_index_length];
|
||||||
|
};
|
||||||
|
typedef struct xdb_vector_index xdb_vector_index_t;
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index(FILE *);
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index_from_file(const char *);
|
||||||
|
|
||||||
|
XDB_PUBLIC(void) xdb_close_vector_index(void *);
|
||||||
|
|
||||||
|
|
||||||
|
// --- content buffer
|
||||||
|
struct xdb_content {
|
||||||
|
unsigned int length;
|
||||||
|
char *buffer;
|
||||||
|
};
|
||||||
|
typedef struct xdb_content xdb_content_t;
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_content_t *) xdb_load_content(FILE *);
|
||||||
|
|
||||||
|
XDB_PUBLIC(xdb_content_t *) xdb_load_content_from_file(const char *);
|
||||||
|
|
||||||
|
XDB_PUBLIC(void) xdb_close_content(void *);
|
||||||
|
|
||||||
|
// --- End buffer load
|
||||||
|
|
||||||
|
// xdb searcher structure
|
||||||
|
struct xdb_searcher_entry {
|
||||||
|
FILE *handle;
|
||||||
|
|
||||||
|
// header info
|
||||||
|
const char *header;
|
||||||
|
int io_count;
|
||||||
|
|
||||||
|
// vector index buffer cache.
|
||||||
|
// preload the vector index will reduce the number of IO operations
|
||||||
|
// thus speedup the search process.
|
||||||
|
const xdb_vector_index_t *v_index;
|
||||||
|
|
||||||
|
// content buffer.
|
||||||
|
// cache the whole xdb content.
|
||||||
|
const xdb_content_t *content;
|
||||||
|
};
|
||||||
|
typedef struct xdb_searcher_entry xdb_searcher_t;
|
||||||
|
|
||||||
|
// xdb searcher new api define
|
||||||
|
XDB_PUBLIC(int) xdb_new_with_file_only(xdb_searcher_t *, const char *);
|
||||||
|
|
||||||
|
XDB_PUBLIC(int) xdb_new_with_vector_index(xdb_searcher_t *, const char *, const xdb_vector_index_t *);
|
||||||
|
|
||||||
|
XDB_PUBLIC(int) xdb_new_with_buffer(xdb_searcher_t *, const xdb_content_t *);
|
||||||
|
|
||||||
|
XDB_PUBLIC(void) xdb_close(void *);
|
||||||
|
|
||||||
|
// xdb searcher search api define
|
||||||
|
XDB_PUBLIC(int) xdb_search_by_string(xdb_searcher_t *, const char *, char *, size_t);
|
||||||
|
|
||||||
|
XDB_PUBLIC(int) xdb_search(xdb_searcher_t *, unsigned int, char *, size_t);
|
||||||
|
|
||||||
|
XDB_PUBLIC(int) xdb_get_io_count(xdb_searcher_t *);
|
||||||
|
|
||||||
|
|
||||||
|
// get unsigned long (4bytes) from a specified buffer start from the specified offset with little-endian
|
||||||
|
XDB_PUBLIC(unsigned int) xdb_get_uint(const char *, int);
|
||||||
|
|
||||||
|
// get unsigned short (2bytes) from a specified buffer start from the specified offset with little-endian
|
||||||
|
XDB_PUBLIC(int) xdb_get_ushort(const char *, int);
|
||||||
|
|
||||||
|
// check the specified string ip and convert it to an unsigned int
|
||||||
|
XDB_PUBLIC(int) xdb_check_ip(const char *, unsigned int *);
|
||||||
|
|
||||||
|
// unsigned int ip to string ip
|
||||||
|
XDB_PUBLIC(void) xdb_long2ip(unsigned int, char *);
|
||||||
|
|
||||||
|
// get the middle ip of a and b
|
||||||
|
XDB_PUBLIC(unsigned int) xdb_mip(unsigned long, unsigned long);
|
||||||
|
|
||||||
|
// get the current time in microseconds
|
||||||
|
XDB_PUBLIC(long) xdb_now();
|
||||||
|
|
||||||
|
|
||||||
|
#endif //C_XDB_SEARCHER_H
|
||||||
|
|
@ -1,525 +0,0 @@
|
||||||
// Copyright 2022 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
|
|
||||||
// ---
|
|
||||||
// @Author Lion <chenxin619315@gmail.com>
|
|
||||||
// @Date 2022/06/27
|
|
||||||
|
|
||||||
#include "xdb_api.h"
|
|
||||||
#include <ctype.h>
|
|
||||||
|
|
||||||
// for Linux
|
|
||||||
#ifdef XDB_LINUX
|
|
||||||
#include "sys/time.h"
|
|
||||||
#endif
|
|
||||||
|
|
||||||
#ifdef XDB_WINDOWS
|
|
||||||
#include <time.h>
|
|
||||||
#endif
|
|
||||||
|
|
||||||
// @Note: since 2023/10/13 to compatible with the windows system
|
|
||||||
#ifdef XDB_WINDOWS
|
|
||||||
static int winsock_initialized = 0;
|
|
||||||
XDB_PUBLIC(int) xdb_init_winsock() {
|
|
||||||
if (winsock_initialized == 1) {
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
WSADATA wsaData;
|
|
||||||
if (WSAStartup(MAKEWORD(2,2), &wsaData) != 0) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
winsock_initialized = 1;
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_clean_winsock() {
|
|
||||||
if (winsock_initialized == 1) {
|
|
||||||
WSACleanup();
|
|
||||||
winsock_initialized = 0;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PRIVATE(int) gettimeofday(struct timeval* tp, void* tzp) {
|
|
||||||
time_t clock;
|
|
||||||
struct tm tm;
|
|
||||||
SYSTEMTIME wtm;
|
|
||||||
GetLocalTime(&wtm);
|
|
||||||
tm.tm_year = wtm.wYear - 1900;
|
|
||||||
tm.tm_mon = wtm.wMonth - 1;
|
|
||||||
tm.tm_mday = wtm.wDay;
|
|
||||||
tm.tm_hour = wtm.wHour;
|
|
||||||
tm.tm_min = wtm.wMinute;
|
|
||||||
tm.tm_sec = wtm.wSecond;
|
|
||||||
tm.tm_isdst = -1;
|
|
||||||
clock = mktime(&tm);
|
|
||||||
tp->tv_sec = clock;
|
|
||||||
tp->tv_usec = wtm.wMilliseconds * 1000;
|
|
||||||
return (0);
|
|
||||||
}
|
|
||||||
#else
|
|
||||||
XDB_PUBLIC(int) xdb_init_winsock() {return 0;}
|
|
||||||
XDB_PUBLIC(void) xdb_clean_winsock() {}
|
|
||||||
#endif
|
|
||||||
|
|
||||||
// --- xdb buffer function implementations
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_header_t *) xdb_load_header(FILE *handle) {
|
|
||||||
xdb_header_t *header;
|
|
||||||
unsigned int size = xdb_header_info_length;
|
|
||||||
|
|
||||||
// entry alloc
|
|
||||||
header = (xdb_header_t *) xdb_malloc(sizeof(xdb_header_t));
|
|
||||||
if (header == NULL) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (fseek(handle, 0, SEEK_SET) == -1) {
|
|
||||||
xdb_free(header);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (fread(header->buffer, 1,size, handle) != size) {
|
|
||||||
xdb_free(header);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
// fill the fields
|
|
||||||
header->length = size;
|
|
||||||
header->version = (unsigned short) xdb_le_get_uint16(header->buffer, 0);
|
|
||||||
header->index_policy = (unsigned short) xdb_le_get_uint16(header->buffer, 2);
|
|
||||||
header->created_at = xdb_le_get_uint32(header->buffer, 4);
|
|
||||||
header->start_index_ptr = xdb_le_get_uint32(header->buffer, 8);
|
|
||||||
header->end_index_ptr = xdb_le_get_uint32(header->buffer,12);
|
|
||||||
|
|
||||||
// since IPv6 supporting
|
|
||||||
header->ip_version = xdb_le_get_uint16(header->buffer, 16);
|
|
||||||
header->runtime_ptr_bytes = xdb_le_get_uint16(header->buffer, 18);
|
|
||||||
|
|
||||||
return header;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_header_t *) xdb_load_header_from_file(const char *db_path) {
|
|
||||||
xdb_header_t *header;
|
|
||||||
FILE *handle = fopen(db_path, "rb");
|
|
||||||
if (handle == NULL) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
header = xdb_load_header(handle);
|
|
||||||
fclose(handle);
|
|
||||||
return header;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_free_header(void *ptr) {
|
|
||||||
xdb_header_t *header = (xdb_header_t *) ptr;
|
|
||||||
if (header->length > 0) {
|
|
||||||
header->length = 0;
|
|
||||||
xdb_free(header);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- vector index
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index(FILE *handle) {
|
|
||||||
xdb_vector_index_t *v_index;
|
|
||||||
unsigned int size = xdb_vector_index_length;
|
|
||||||
|
|
||||||
// seek to the vector index offset
|
|
||||||
if (fseek(handle, xdb_header_info_length, SEEK_SET) == -1) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
// do the buffer read
|
|
||||||
v_index = (xdb_vector_index_t *) xdb_malloc(sizeof(xdb_vector_index_t));
|
|
||||||
if (v_index == NULL) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
v_index->length = size;
|
|
||||||
if (fread(v_index->buffer, 1, size, handle) != size) {
|
|
||||||
xdb_free(v_index);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
return v_index;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index_from_file(const char *db_path) {
|
|
||||||
xdb_vector_index_t *v_index;
|
|
||||||
FILE *handle = fopen(db_path, "rb");
|
|
||||||
if (handle == NULL) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
v_index = xdb_load_vector_index(handle);
|
|
||||||
fclose(handle);
|
|
||||||
return v_index;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_free_vector_index(void *ptr) {
|
|
||||||
xdb_vector_index_t *v_index = (xdb_vector_index_t *) ptr;
|
|
||||||
if (v_index->length > 0) {
|
|
||||||
v_index->length = 0;
|
|
||||||
xdb_free(v_index);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- content buffer
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_content_t *) xdb_load_content(FILE *handle) {
|
|
||||||
unsigned int size;
|
|
||||||
xdb_content_t *content;
|
|
||||||
|
|
||||||
// determine the file size
|
|
||||||
if (fseek(handle, 0, SEEK_END) == -1) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
size = (unsigned int) ftell(handle);
|
|
||||||
if (fseek(handle, 0, SEEK_SET) == -1) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
// do the file read
|
|
||||||
content = (xdb_content_t *) xdb_malloc(sizeof(xdb_content_t));
|
|
||||||
if (content == NULL) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
// do the buffer alloc
|
|
||||||
content->buffer = (char *) xdb_malloc(size);
|
|
||||||
if (content->buffer == NULL) {
|
|
||||||
xdb_free(content);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
// read the content into the buffer
|
|
||||||
content->length = size;
|
|
||||||
if (fread(content->buffer, 1, size, handle) != size) {
|
|
||||||
xdb_free(content);
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
return content;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_content_t *) xdb_load_content_from_file(const char *db_path) {
|
|
||||||
xdb_content_t *content;
|
|
||||||
FILE *handle = fopen(db_path, "rb");
|
|
||||||
if (handle == NULL) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
content = xdb_load_content(handle);
|
|
||||||
fclose(handle);
|
|
||||||
return content;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(void) xdb_free_content(void *ptr) {
|
|
||||||
xdb_content_t *content = (xdb_content_t *) ptr;
|
|
||||||
if (content->length > 0) {
|
|
||||||
content->length = 0;
|
|
||||||
xdb_free(content->buffer);
|
|
||||||
content->buffer = NULL;
|
|
||||||
xdb_free(content);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_verify_from_header(FILE *handle, xdb_header_t *header) {
|
|
||||||
unsigned int runtime_ptr_bytes = 0; // runtime ptr bytes
|
|
||||||
if (header->version == xdb_structure_20) {
|
|
||||||
runtime_ptr_bytes = 4;
|
|
||||||
} else if (header->version == xdb_structure_30) {
|
|
||||||
runtime_ptr_bytes = header->runtime_ptr_bytes;
|
|
||||||
} else {
|
|
||||||
return 2;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 1, confirm the xdb file size.
|
|
||||||
// to ensure that the maximum file pointer does not overflow.
|
|
||||||
int err = fseek(handle, 0L, SEEK_END);
|
|
||||||
if (err != 0) {
|
|
||||||
return 3;
|
|
||||||
}
|
|
||||||
|
|
||||||
long long fileBytes = xdb_ftell(handle);
|
|
||||||
long long maxFilePtr = (1LL << (runtime_ptr_bytes * 8)) - 1;
|
|
||||||
// printf("fileBytes: %lld, maxFilePtr: %lld\n", fileBytes, maxFilePtr);
|
|
||||||
if (fileBytes > maxFilePtr) {
|
|
||||||
return 4;
|
|
||||||
}
|
|
||||||
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_verify(FILE *handle) {
|
|
||||||
xdb_header_t *header = xdb_load_header(handle);
|
|
||||||
if (header == NULL) {
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
int errcode = xdb_verify_from_header(handle, header);
|
|
||||||
if (errcode != 0) {
|
|
||||||
goto done;
|
|
||||||
}
|
|
||||||
|
|
||||||
// what next ?
|
|
||||||
done:
|
|
||||||
xdb_free_header(header);
|
|
||||||
return errcode;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_verify_from_file(const char *db_path) {
|
|
||||||
FILE *handle = fopen(db_path, "rb");
|
|
||||||
if (handle == NULL) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
int r = xdb_verify(handle);
|
|
||||||
fclose(handle);
|
|
||||||
return r;
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- End content buffer
|
|
||||||
|
|
||||||
|
|
||||||
// --- ip version
|
|
||||||
|
|
||||||
// ip compare for IPv4
|
|
||||||
// ip1 - with Big endian byte order parsed from an input
|
|
||||||
// ip2 - with Little endian byte order read from the xdb index.
|
|
||||||
// to compatiable with the Little Endian encoded IPv4 on xdb 2.0.
|
|
||||||
XDB_PRIVATE(int) _ipv4_sub_compare(const bytes_ip_t *ip_bytes, int bytes, const char *buffer, int offset) {
|
|
||||||
register int i0, i1;
|
|
||||||
for (int i = 0, j = offset + bytes - 1; i < bytes; i++, j--) {
|
|
||||||
i0 = ip_bytes[i];
|
|
||||||
i1 = buffer[j] & 0xFF;
|
|
||||||
if (i0 > i1) {
|
|
||||||
return 1;
|
|
||||||
} else if (i0 < i1) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
static xdb_version_t _ip_version_list[] = {
|
|
||||||
// 14 = 4 + 4 + 2 + 4
|
|
||||||
{xdb_ipv4_id, "IPv4", xdb_ipv4_bytes, xdb_v4_index_size, _ipv4_sub_compare},
|
|
||||||
|
|
||||||
// 38 = 16 + 16 + 2 + 4
|
|
||||||
{xdb_ipv6_id, "IPv6", xdb_ipv6_bytes, xdb_v6_index_size, xdb_ip_sub_compare},
|
|
||||||
|
|
||||||
// END
|
|
||||||
{0, NULL, 0, 0, NULL}
|
|
||||||
};
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_version_v4() {
|
|
||||||
return &_ip_version_list[0];
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_version_v6() {
|
|
||||||
return &_ip_version_list[1];
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_version_is_v4(const xdb_version_t *version) {
|
|
||||||
return version->id == xdb_ipv4_id;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_version_is_v6(const xdb_version_t *version) {
|
|
||||||
return version->id == xdb_ipv6_id;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_version_from_name(char *name) {
|
|
||||||
// to upper case the name
|
|
||||||
for (int i = 0; name[i] != '\0'; i++) {
|
|
||||||
name[i] = toupper((unsigned char) name[i]);
|
|
||||||
}
|
|
||||||
|
|
||||||
if (strcmp(name, "V4") == 0 || strcmp(name, "IPV4") == 0) {
|
|
||||||
return xdb_version_v4();
|
|
||||||
} else if (strcmp(name, "V6") == 0 || strcmp(name, "IPV6") == 0) {
|
|
||||||
return xdb_version_v6();
|
|
||||||
} else {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_version_from_header(xdb_header_t *header) {
|
|
||||||
// Old structure with ONLY IPv4 supports
|
|
||||||
if (header->version == xdb_structure_20) {
|
|
||||||
return xdb_version_v4();
|
|
||||||
}
|
|
||||||
|
|
||||||
// structure 3.0 with IPv6 supporting
|
|
||||||
if (header->version != xdb_structure_30) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (header->ip_version == xdb_ipv4_id) {
|
|
||||||
return xdb_version_v4();
|
|
||||||
} else if (header->ip_version == xdb_ipv6_id) {
|
|
||||||
return xdb_version_v6();
|
|
||||||
} else {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- END ip version
|
|
||||||
|
|
||||||
XDB_PUBLIC(long) xdb_now() {
|
|
||||||
struct timeval c_time;
|
|
||||||
gettimeofday(&c_time, NULL);
|
|
||||||
return c_time.tv_sec * (int)1e6 + c_time.tv_usec;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(unsigned int) xdb_le_get_uint32(const char *buffer, int offset) {
|
|
||||||
return (
|
|
||||||
((buffer[offset ]) & 0x000000FF) |
|
|
||||||
((buffer[offset+1] << 8) & 0x0000FF00) |
|
|
||||||
((buffer[offset+2] << 16) & 0x00FF0000) |
|
|
||||||
((buffer[offset+3] << 24) & 0xFF000000)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_le_get_uint16(const char *buffer, int offset) {
|
|
||||||
return (
|
|
||||||
((buffer[offset ]) & 0x000000FF) |
|
|
||||||
((buffer[offset+1] << 8) & 0x0000FF00)
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_parse_ip(const string_ip_t *ip_string, bytes_ip_t *buffer, size_t length) {
|
|
||||||
char *d_ptr = strchr(ip_string, '.');
|
|
||||||
char *c_ptr = strchr(ip_string, ':');
|
|
||||||
// version check
|
|
||||||
if (d_ptr != NULL && c_ptr == NULL) {
|
|
||||||
return xdb_parse_v4_ip(ip_string, buffer, length);
|
|
||||||
} else if (c_ptr != NULL) {
|
|
||||||
return xdb_parse_v6_ip(ip_string, buffer, length);
|
|
||||||
}
|
|
||||||
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_parse_v4_ip(const string_ip_t *ip_string, bytes_ip_t *buffer, size_t length) {
|
|
||||||
struct in_addr addr;
|
|
||||||
|
|
||||||
// buffer length checking
|
|
||||||
if (length < xdb_ipv4_bytes) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (inet_pton(AF_INET, ip_string, &addr) != 1) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
// encode the address to buffer with big endian byte bufffer.
|
|
||||||
buffer[0] = (addr.s_addr) & 0xFF;
|
|
||||||
buffer[1] = (addr.s_addr >> 8) & 0xFF;
|
|
||||||
buffer[2] = (addr.s_addr >> 16) & 0xFF;
|
|
||||||
buffer[3] = (addr.s_addr >> 24) & 0xFF;
|
|
||||||
return XDB_IPv4;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(xdb_version_t *) xdb_parse_v6_ip(const string_ip_t *ip_string, bytes_ip_t *buffer, size_t length) {
|
|
||||||
struct in6_addr addr;
|
|
||||||
|
|
||||||
// buffer length checking
|
|
||||||
if (length < xdb_ipv6_bytes) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (inet_pton(AF_INET6, ip_string, &addr) != 1) {
|
|
||||||
return NULL;
|
|
||||||
}
|
|
||||||
|
|
||||||
memcpy(buffer, addr.s6_addr, xdb_ipv6_bytes);
|
|
||||||
return XDB_IPv6;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_ip_to_string(const bytes_ip_t *ip_bytes, int bytes, char *ip_string, size_t length) {
|
|
||||||
if (bytes == xdb_ipv4_bytes) {
|
|
||||||
return xdb_v4_ip_to_string(ip_bytes, ip_string, length);
|
|
||||||
} else if (bytes == xdb_ipv6_bytes) {
|
|
||||||
return xdb_v6_ip_to_string(ip_bytes, ip_string, length);
|
|
||||||
}
|
|
||||||
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_v4_ip_to_string(const bytes_ip_t *ip_bytes, char *ip_string, size_t length) {
|
|
||||||
if (!ip_bytes || !ip_string || length == 0) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
// buffer length checking
|
|
||||||
if (length < INET_ADDRSTRLEN) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (inet_ntop(AF_INET, ip_bytes, ip_string, length) == NULL) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_v6_ip_to_string(const bytes_ip_t *ip_bytes, char *ip_string, size_t length) {
|
|
||||||
if (!ip_bytes || !ip_string || length == 0) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (length < INET6_ADDRSTRLEN) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (inet_ntop(AF_INET6, ip_bytes, ip_string, length) == NULL) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_ip_sub_compare(const bytes_ip_t *ip1, int bytes, const char *buffer, int offset) {
|
|
||||||
register int i, i1, i2;
|
|
||||||
for (i = 0; i < bytes; i++) {
|
|
||||||
i1 = ip1[i];
|
|
||||||
i2 = buffer[offset + i] & 0xFF;
|
|
||||||
if (i1 > i2) {
|
|
||||||
return 1;
|
|
||||||
} else if (i1 < i2) {
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(int) xdb_fseek(FILE *handle, long long offset, int whence) {
|
|
||||||
// we may have to use the large file solution later
|
|
||||||
// #if defined(XDB_LINUX)
|
|
||||||
// return fseeko(handle, (off_t) offset, whence);
|
|
||||||
// #elif defined(XDB_WINDOWS)
|
|
||||||
// return _fseeki64(handle, (__int64) offset, whence)
|
|
||||||
// #else
|
|
||||||
// return fseek(handle, (long) offset, whence);
|
|
||||||
// #endif
|
|
||||||
|
|
||||||
return fseek(handle, (long) offset, whence);
|
|
||||||
}
|
|
||||||
|
|
||||||
XDB_PUBLIC(long long) xdb_ftell(FILE *handle) {
|
|
||||||
// we may have to use the large file solution later
|
|
||||||
// #if defined(XDB_LINUX)
|
|
||||||
// return (long long) ftello(handle);
|
|
||||||
// #elif defined(XDB_WINDOWS)
|
|
||||||
// return (long long) _ftelli64(handle);
|
|
||||||
// #else
|
|
||||||
// // report error ?
|
|
||||||
// return (long long) ftell(handle);
|
|
||||||
// #endif
|
|
||||||
|
|
||||||
return (long long) ftell(handle);
|
|
||||||
}
|
|
||||||
|
|
@ -1,5 +0,0 @@
|
||||||
# build cache
|
|
||||||
.cache/
|
|
||||||
|
|
||||||
# dependency lock
|
|
||||||
cjpm.lock
|
|
||||||
|
|
@ -1,172 +0,0 @@
|
||||||
# ip2region Cangjie Query Client
|
|
||||||
|
|
||||||
This implementation supports both `IPv4` and `IPv6`, and provides a `thread-safe` service layer.
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
```
|
|
||||||
binding/cangjie/
|
|
||||||
├── src/
|
|
||||||
│ ├── xdb/ # Low-level xdb query engine
|
|
||||||
│ │ ├── searcher.cj # Searcher (file-only / vector-index / content-buff modes)
|
|
||||||
│ │ ├── util.cj # IP parsing/comparison, byte readers
|
|
||||||
│ │ ├── header.cj # xdb header + version parsing
|
|
||||||
│ │ ├── version.cj # IPv4/IPv6 version definitions
|
|
||||||
│ │ ├── *_test.cj # Unit tests (24 cases)
|
|
||||||
│ ├── service/ # High-level thread-safe service layer
|
|
||||||
│ │ ├── config.cj # Config (cache policy, pool size, pre-loaded data)
|
|
||||||
│ │ ├── searcher_pool.cj # SearcherPool (Semaphore + ConcurrentLinkedQueue)
|
|
||||||
│ │ ├── ip2region.cj # Ip2Region unified API
|
|
||||||
│ │ ├── *_test.cj # Unit tests (17 cases)
|
|
||||||
├── example/ # Executable CLI demo (separate project)
|
|
||||||
│ ├── cjpm.toml
|
|
||||||
│ └── src/main.cj # 5 usage demos + benchmarks
|
|
||||||
├── cjpm.toml # Static library project
|
|
||||||
└── README.md
|
|
||||||
```
|
|
||||||
|
|
||||||
**Two API layers:**
|
|
||||||
|
|
||||||
- **`ip2region.xdb`** — Low-level Searcher. Not thread-safe (mutable `ioCount` + `File` seek/read state). Supports three cache modes.
|
|
||||||
- **`ip2region.service`** — High-level `Ip2Region` class wrapping `SearcherPool`. Thread-safe, handles v4/v6 dispatch automatically.
|
|
||||||
|
|
||||||
## Build
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd binding/cangjie
|
|
||||||
cjpm build # static library
|
|
||||||
cd example
|
|
||||||
cjpm build # example executable
|
|
||||||
```
|
|
||||||
|
|
||||||
## Test
|
|
||||||
|
|
||||||
41 tests across both packages (24 xdb + 17 service):
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd binding/cangjie
|
|
||||||
cjpm test
|
|
||||||
```
|
|
||||||
|
|
||||||
## Cache Policies
|
|
||||||
|
|
||||||
| Policy | Memory | Speed | Thread-safe* |
|
|
||||||
|--------|--------|-------|-------------|
|
|
||||||
| `FileOnly` (0) | ~0 MB | ~21 µs/op (v4) | Via SearcherPool |
|
|
||||||
| `VectorIndex` (1) | ~4 MB | ~20 µs/op (v4) | Via SearcherPool |
|
|
||||||
| `ContentBuff` (2) | ~full xdb | ~5 µs/op (v4) | Via SearcherPool |
|
|
||||||
|
|
||||||
\* Low-level `Searcher` is NOT thread-safe. The service layer (`SearcherPool` / `Ip2Region`) provides thread safety by pooling searchers (FileOnly/VectorIndex) or sharing a read-only buffer (ContentBuff).
|
|
||||||
|
|
||||||
## Benchmark Results
|
|
||||||
|
|
||||||
Benchmarked on Windows 11, AMD Ryzen 7, 487k IPv4 records / 638k IPv6 records:
|
|
||||||
|
|
||||||
### IPv4 (`ip2region_v4.xdb`, 487,167 records)
|
|
||||||
|
|
||||||
| Cache Policy | Total | Time | Avg |
|
|
||||||
|-------------|-------|------|-----|
|
|
||||||
| ContentBuff | 487,167 | 3,059 ms | **5 µs/op** |
|
|
||||||
| VectorIndex | 487,167 | 10,413 ms | **20 µs/op** |
|
|
||||||
| FileOnly | 487,167 | 11,271 ms | **21 µs/op** |
|
|
||||||
|
|
||||||
### IPv6 (`ip2region_v6.xdb`, 638,953 records)
|
|
||||||
|
|
||||||
| Cache Policy | Total | Time | Avg |
|
|
||||||
|-------------|-------|------|-----|
|
|
||||||
| ContentBuff | 638,953 | 37,080 ms | **56 µs/op** |
|
|
||||||
| VectorIndex | 638,953 | 38,443 ms | **58 µs/op** |
|
|
||||||
| FileOnly | 638,953 | 19,148 ms | **29 µs/op** |
|
|
||||||
|
|
||||||
Run your own:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd example
|
|
||||||
cjpm run -- bench --db ../../../data/ip2region_v4.xdb --src ../../../data/ipv4_source.txt --cache-policy content
|
|
||||||
cjpm run -- bench --db ../../../data/ip2region_v6.xdb --src ../../../data/ipv6_source.txt --cache-policy file
|
|
||||||
```
|
|
||||||
|
|
||||||
## Usage
|
|
||||||
|
|
||||||
### Low-level API (`ip2region.xdb`)
|
|
||||||
|
|
||||||
```cangjie
|
|
||||||
import ip2region.xdb.*
|
|
||||||
|
|
||||||
// Content buffer mode (fastest)
|
|
||||||
let content = File.readFrom(Path("ip2region_v4.xdb"))
|
|
||||||
let header = newHeaderFromBytes(content)
|
|
||||||
let version = versionFromHeader(header)
|
|
||||||
let searcher = Searcher(version, content, "ip2region_v4.xdb")
|
|
||||||
let region = searcher.search(parseIP("220.181.108.183"))
|
|
||||||
searcher.close()
|
|
||||||
|
|
||||||
// Vector index mode
|
|
||||||
let vIndex = loadVectorIndex(content)
|
|
||||||
let searcher2 = Searcher(version, "ip2region_v4.xdb", vIndex)
|
|
||||||
searcher2.close()
|
|
||||||
|
|
||||||
// File only mode
|
|
||||||
let searcher3 = Searcher(version, "ip2region_v4.xdb")
|
|
||||||
searcher3.close()
|
|
||||||
```
|
|
||||||
|
|
||||||
### High-level API (`ip2region.service`)
|
|
||||||
|
|
||||||
```cangjie
|
|
||||||
import ip2region.service.*
|
|
||||||
|
|
||||||
// Via factory (VectorIndex, 20 searchers)
|
|
||||||
let region = newIp2Region("ip2region_v4.xdb", "ip2region_v6.xdb")
|
|
||||||
let result = region.search("220.181.108.183")
|
|
||||||
region.close()
|
|
||||||
|
|
||||||
// Custom config
|
|
||||||
let v4Cfg = Config(ContentBuff, IPv4, "ip2region_v4.xdb", 10)
|
|
||||||
let v6Cfg = Config(VectorIndex, IPv6, "ip2region_v6.xdb", 20)
|
|
||||||
let svc = Ip2Region(v4Cfg, v6Cfg)
|
|
||||||
let result = svc.search("2408:8266:100:1000::")
|
|
||||||
svc.close()
|
|
||||||
```
|
|
||||||
|
|
||||||
### Single-version API
|
|
||||||
|
|
||||||
```cangjie
|
|
||||||
import ip2region.service.*
|
|
||||||
|
|
||||||
// IPv4 only — Ip2Region(config, IPv4)
|
|
||||||
let v4Cfg = Config(ContentBuff, IPv4, "ip2region_v4.xdb", 10)
|
|
||||||
let v4 = Ip2Region(v4Cfg, IPv4)
|
|
||||||
let r4 = v4.search("220.181.108.183")
|
|
||||||
v4.close()
|
|
||||||
|
|
||||||
// IPv6 only — Ip2Region(config, IPv6)
|
|
||||||
let v6Cfg = Config(ContentBuff, IPv6, "ip2region_v6.xdb", 10)
|
|
||||||
let v6 = Ip2Region(v6Cfg, IPv6)
|
|
||||||
let r6 = v6.search("2408:8266:100:1000::")
|
|
||||||
v6.close()
|
|
||||||
|
|
||||||
// Factory functions
|
|
||||||
let v4Only = newIp2RegionV4(v4Cfg)
|
|
||||||
let v6Only = newIp2RegionV6(v6Cfg)
|
|
||||||
v4Only.close(); v6Only.close()
|
|
||||||
```
|
|
||||||
|
|
||||||
### CLI
|
|
||||||
|
|
||||||
```bash
|
|
||||||
cd example
|
|
||||||
cjpm run
|
|
||||||
```
|
|
||||||
|
|
||||||
## Thread Safety
|
|
||||||
|
|
||||||
- **`xdb.Searcher`** — NOT thread-safe. Keep one per thread or use a pool.
|
|
||||||
- **`service.SearcherPool`** — Thread-safe. Pre-allocates N searchers, uses `Semaphore` for backpressure and `ConcurrentLinkedQueue` for storage. `borrow()` blocks until a searcher is available.
|
|
||||||
- **`service.Ip2Region`** — Thread-safe. Delegates to pool or shared in-mem searcher based on cache policy. ContentBuff mode shares a single Searcher (Array<Byte> is read-only), FileOnly/VectorIndex use the pool.
|
|
||||||
- **`service.Config`** — Immutable after construction. Safe to share.
|
|
||||||
|
|
||||||
## Requirements
|
|
||||||
|
|
||||||
- Cangjie SDK 1.1.0+
|
|
||||||
- `cjpm` build tool
|
|
||||||
|
|
@ -1,15 +0,0 @@
|
||||||
[package]
|
|
||||||
cjc-version = "1.1.0"
|
|
||||||
name = "ip2region"
|
|
||||||
organization = ""
|
|
||||||
description = "ip2region xdb searcher library for Cangjie"
|
|
||||||
version = "1.0.0"
|
|
||||||
target-dir = ""
|
|
||||||
script-dir = ""
|
|
||||||
output-type = "static"
|
|
||||||
compile-option = "-Woff unused"
|
|
||||||
override-compile-option = ""
|
|
||||||
link-option = ""
|
|
||||||
package-configuration = {}
|
|
||||||
|
|
||||||
[dependencies]
|
|
||||||
|
|
@ -1,10 +0,0 @@
|
||||||
[package]
|
|
||||||
cjc-version = "1.1.0"
|
|
||||||
name = "ip2regionExample"
|
|
||||||
version = "1.0.0"
|
|
||||||
output-type = "executable"
|
|
||||||
compile-option = "-Woff unused"
|
|
||||||
description = "ip2region Cangjie CLI example"
|
|
||||||
|
|
||||||
[dependencies]
|
|
||||||
ip2region = { path = ".." }
|
|
||||||
|
|
@ -1,196 +0,0 @@
|
||||||
package ip2regionExample
|
|
||||||
|
|
||||||
import std.fs.*
|
|
||||||
import std.time.*
|
|
||||||
import std.collection.*
|
|
||||||
import ip2region.xdb.*
|
|
||||||
import ip2region.service.*
|
|
||||||
|
|
||||||
// ip2region Cangjie 使用示例
|
|
||||||
//
|
|
||||||
// 构建: cjpm build
|
|
||||||
// 运行: cjpm run
|
|
||||||
//
|
|
||||||
// 默认从 ../../../data/ 查找 xdb 文件，
|
|
||||||
// 也可通过命令行参数指定路径:
|
|
||||||
// cjpm run -- --v4-db <path> --v6-db <path>
|
|
||||||
|
|
||||||
func printHelp() {
|
|
||||||
println("ip2region Cangjie 使用示例")
|
|
||||||
println("用法: cjpm run -- [options]")
|
|
||||||
println("选项:")
|
|
||||||
println(" --v4-db <path> IPv4 xdb 文件路径 (默认: ../../../data/ip2region_v4.xdb)")
|
|
||||||
println(" --v6-db <path> IPv6 xdb 文件路径 (默认: ../../../data/ip2region_v6.xdb)")
|
|
||||||
println(" --help 显示此帮助")
|
|
||||||
}
|
|
||||||
|
|
||||||
// ============================================================
|
|
||||||
// 方式一: 使用底层 Searcher API (ip2region.xdb)
|
|
||||||
// 可自行选择缓存策略，单线程使用
|
|
||||||
// ============================================================
|
|
||||||
func demoXdbSearcher(v4Path: String, v6Path: String) {
|
|
||||||
println("\n=== 方式一: 底层 Searcher API ===")
|
|
||||||
|
|
||||||
// 1. 读取 xdb 文件到内存
|
|
||||||
let v4Content = File.readFrom(Path(v4Path))
|
|
||||||
|
|
||||||
// 2. 解析文件头，获取版本信息
|
|
||||||
let v4Header = newHeaderFromBytes(v4Content)
|
|
||||||
let v4Version = versionFromHeader(v4Header)
|
|
||||||
|
|
||||||
// 3. 选择缓存策略创建 Searcher
|
|
||||||
// 支持三种模式: ContentBuff(最快), VectorIndex(均衡), FileOnly(省内存)
|
|
||||||
let vIndex = loadVectorIndex(v4Content)
|
|
||||||
let searcher = Searcher(v4Version, v4Path, vIndex) // VectorIndex 模式
|
|
||||||
|
|
||||||
// 4. 查询 IP
|
|
||||||
let ip = parseIP("220.181.108.183")
|
|
||||||
let region = searcher.search(ip)
|
|
||||||
println(" IP: 220.181.108.183")
|
|
||||||
println(" 结果: ${region}")
|
|
||||||
|
|
||||||
searcher.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
// ============================================================
|
|
||||||
// 方式二: 使用 Ip2Region 服务 API (ip2region.service)
|
|
||||||
// 自动管理连接池，支持多线程安全访问
|
|
||||||
// ============================================================
|
|
||||||
func demoIp2Region(v4Path: String, v6Path: String) {
|
|
||||||
println("\n=== 方式二: Ip2Region 服务 API ===")
|
|
||||||
|
|
||||||
// 1. 创建配置 (缓存策略 + 连接池大小)
|
|
||||||
let v4Cfg = Config(VectorIndex, IPv4, v4Path, 10)
|
|
||||||
let v6Cfg = Config(VectorIndex, IPv6, v6Path, 10)
|
|
||||||
|
|
||||||
// 2. 创建服务 (同时支持 IPv4 + IPv6)
|
|
||||||
let region = Ip2Region(v4Cfg, v6Cfg)
|
|
||||||
|
|
||||||
// 3. 查询 IPv4
|
|
||||||
let r1 = region.search("220.181.108.183")
|
|
||||||
println(" IPv4: 220.181.108.183 -> ${r1}")
|
|
||||||
|
|
||||||
// 4. 查询 IPv6
|
|
||||||
let r2 = region.search("2408:8266:100:1000::")
|
|
||||||
println(" IPv6: 2408:8266:100:1000:: -> ${r2}")
|
|
||||||
|
|
||||||
region.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
// ============================================================
|
|
||||||
// 方式三: 单版本服务 (仅 IPv4 或 仅 IPv6)
|
|
||||||
// 使用 init(config, ipVersion) 构造函数
|
|
||||||
// ============================================================
|
|
||||||
func demoSingleVersion(v4Path: String, v6Path: String) {
|
|
||||||
println("\n=== 方式三: 单版本服务 ===")
|
|
||||||
|
|
||||||
// IPv4 only
|
|
||||||
let v4Cfg = Config(ContentBuff, IPv4, v4Path, 3)
|
|
||||||
let v4Region = Ip2Region(v4Cfg, IPv4)
|
|
||||||
let r4 = v4Region.search("220.181.108.183")
|
|
||||||
println(" IPv4 only: 220.181.108.183 -> ${r4}")
|
|
||||||
v4Region.close()
|
|
||||||
|
|
||||||
// IPv6 only
|
|
||||||
let v6Cfg = Config(ContentBuff, IPv6, v6Path, 3)
|
|
||||||
let v6Region = Ip2Region(v6Cfg, IPv6)
|
|
||||||
let r6 = v6Region.search("2408:8266:100:1000::")
|
|
||||||
println(" IPv6 only: 2408:8266:100:1000:: -> ${r6}")
|
|
||||||
v6Region.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
// ============================================================
|
|
||||||
// 方式四: 使用简便工厂函数 (ip2region.service)
|
|
||||||
// 一行创建，默认 VectorIndex + 20 连接池
|
|
||||||
// ============================================================
|
|
||||||
func demoFactory(v4Path: String, v6Path: String) {
|
|
||||||
println("\n=== 方式四: 简便工厂函数 ===")
|
|
||||||
|
|
||||||
let region = newIp2Region(v4Path, v6Path)
|
|
||||||
let result = region.search("220.181.108.183")
|
|
||||||
println(" 220.181.108.183 -> ${result}")
|
|
||||||
region.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
// ============================================================
|
|
||||||
// 性能测试: 测试三种缓存策略的查询速度
|
|
||||||
// ============================================================
|
|
||||||
func bench(v4Path: String) {
|
|
||||||
println("\n=== 性能测试 (IPv4, 487k 条记录) ===")
|
|
||||||
|
|
||||||
let content = File.readFrom(Path(v4Path))
|
|
||||||
let header = newHeaderFromBytes(content)
|
|
||||||
let version = versionFromHeader(header)
|
|
||||||
let vIndex = loadVectorIndex(content)
|
|
||||||
|
|
||||||
// 从数据文件读取测试 IP 列表
|
|
||||||
let srcPath = v4Path.replace("ip2region_v4.xdb", "ipv4_source.txt")
|
|
||||||
let srcContent = File.readFrom(Path(srcPath))
|
|
||||||
let lines = String.fromUtf8(srcContent).split("\n")
|
|
||||||
|
|
||||||
var ipList = ArrayList<Array<Byte>>()
|
|
||||||
for (line in lines) {
|
|
||||||
let trimmed = line.trimAsciiStart().trimAsciiEnd()
|
|
||||||
if (trimmed.size == 0) { continue }
|
|
||||||
let parts = trimmed.split("|")
|
|
||||||
if (parts.size < 3) { continue }
|
|
||||||
try { ipList.add(parseIP(parts[0])) } catch (_) { continue }
|
|
||||||
}
|
|
||||||
println(" 加载了 ${ipList.size} 个 IP")
|
|
||||||
|
|
||||||
// 测试 ContentBuff 模式
|
|
||||||
println("\n [ContentBuff 模式]")
|
|
||||||
let s1 = Searcher(version, content, v4Path)
|
|
||||||
benchSearcher(s1, ipList)
|
|
||||||
|
|
||||||
// 测试 VectorIndex 模式
|
|
||||||
println(" [VectorIndex 模式]")
|
|
||||||
let s2 = Searcher(version, v4Path, vIndex)
|
|
||||||
benchSearcher(s2, ipList)
|
|
||||||
|
|
||||||
// 测试 FileOnly 模式
|
|
||||||
println(" [FileOnly 模式]")
|
|
||||||
let s3 = Searcher(version, v4Path)
|
|
||||||
benchSearcher(s3, ipList)
|
|
||||||
|
|
||||||
s1.close(); s2.close(); s3.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
func benchSearcher(searcher: Searcher, ipList: ArrayList<Array<Byte>>) {
|
|
||||||
let count = ipList.size
|
|
||||||
let start = MonoTime.now()
|
|
||||||
for (i in 0..count) {
|
|
||||||
let _ = searcher.search(ipList[i])
|
|
||||||
}
|
|
||||||
let elapsed = MonoTime.now() - start
|
|
||||||
let avg = elapsed.toMicroseconds() / count
|
|
||||||
println(" 总量: ${count}, 耗时: ${elapsed.toMilliseconds()} ms, 平均: ${avg} us/op")
|
|
||||||
}
|
|
||||||
|
|
||||||
main(args: Array<String>): Int64 {
|
|
||||||
var v4Path = "../../../data/ip2region_v4.xdb"
|
|
||||||
var v6Path = "../../../data/ip2region_v6.xdb"
|
|
||||||
|
|
||||||
var i = 0
|
|
||||||
while (i < args.size) {
|
|
||||||
if (args[i] == "--v4-db" && i + 1 < args.size) {
|
|
||||||
v4Path = args[i + 1]; i = i + 2
|
|
||||||
} else if (args[i] == "--v6-db" && i + 1 < args.size) {
|
|
||||||
v6Path = args[i + 1]; i = i + 2
|
|
||||||
} else if (args[i] == "--help") {
|
|
||||||
printHelp(); return 0
|
|
||||||
} else { i = i + 1 }
|
|
||||||
}
|
|
||||||
|
|
||||||
println("ip2region Cangjie 使用示例")
|
|
||||||
println("xdb 路径: ${v4Path}")
|
|
||||||
|
|
||||||
demoXdbSearcher(v4Path, v6Path)
|
|
||||||
demoIp2Region(v4Path, v6Path)
|
|
||||||
demoSingleVersion(v4Path, v6Path)
|
|
||||||
demoFactory(v4Path, v6Path)
|
|
||||||
bench(v4Path)
|
|
||||||
|
|
||||||
println("\n完成!")
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
@ -1,225 +0,0 @@
|
||||||
package ip2region
|
|
||||||
|
|
||||||
import std.fs.*
|
|
||||||
import std.time.*
|
|
||||||
import std.env.*
|
|
||||||
import ip2region.xdb.*
|
|
||||||
|
|
||||||
func printHelp() {
|
|
||||||
println("ip2region xdb searcher")
|
|
||||||
println("Usage: ip2region [command] [command options]")
|
|
||||||
println("Command:")
|
|
||||||
println(" search search input test")
|
|
||||||
println(" bench search bench test")
|
|
||||||
}
|
|
||||||
|
|
||||||
func runSearch(args: Array<String>) {
|
|
||||||
var v4DbPath = "../../data/ip2region_v4.xdb"
|
|
||||||
var v6DbPath = "../../data/ip2region_v6.xdb"
|
|
||||||
var v4CachePolicy = "vectorIndex"
|
|
||||||
var v6CachePolicy = "vectorIndex"
|
|
||||||
var showHelp = false
|
|
||||||
|
|
||||||
var i = 1 // skip command name
|
|
||||||
while (i < args.size) {
|
|
||||||
let arg = args[i]
|
|
||||||
if (arg == "--v4-db" && i + 1 < args.size) {
|
|
||||||
v4DbPath = args[i + 1]
|
|
||||||
i = i + 2
|
|
||||||
} else if (arg == "--v6-db" && i + 1 < args.size) {
|
|
||||||
v6DbPath = args[i + 1]
|
|
||||||
i = i + 2
|
|
||||||
} else if (arg == "--v4-cache-policy" && i + 1 < args.size) {
|
|
||||||
v4CachePolicy = args[i + 1]
|
|
||||||
i = i + 2
|
|
||||||
} else if (arg == "--v6-cache-policy" && i + 1 < args.size) {
|
|
||||||
v6CachePolicy = args[i + 1]
|
|
||||||
i = i + 2
|
|
||||||
} else if (arg == "--help") {
|
|
||||||
showHelp = true
|
|
||||||
i = i + 1
|
|
||||||
} else {
|
|
||||||
i = i + 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (showHelp) {
|
|
||||||
println("ip2region search [command options]")
|
|
||||||
println("options:")
|
|
||||||
println(" --v4-db <path> ip2region v4 binary xdb file path")
|
|
||||||
println(" --v4-cache-policy <mode> v4 cache policy: file/vectorIndex/content")
|
|
||||||
println(" --v6-db <path> ip2region v6 binary xdb file path")
|
|
||||||
println(" --v6-cache-policy <mode> v6 cache policy: file/vectorIndex/content")
|
|
||||||
println(" --help print this help")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
// Load v4 xdb
|
|
||||||
print("Loading v4 xdb ... ")
|
|
||||||
let v4Content = File.readFrom(Path(v4DbPath))
|
|
||||||
let v4Header = newHeaderFromBytes(v4Content)
|
|
||||||
let v4Version = versionFromHeader(v4Header)
|
|
||||||
let v4Searcher = createSearcher(v4Version, v4Content, v4DbPath, v4CachePolicy)
|
|
||||||
println("done")
|
|
||||||
|
|
||||||
// Load v6 xdb
|
|
||||||
print("Loading v6 xdb ... ")
|
|
||||||
let v6Content = File.readFrom(Path(v6DbPath))
|
|
||||||
let v6Header = newHeaderFromBytes(v6Content)
|
|
||||||
let v6Version = versionFromHeader(v6Header)
|
|
||||||
let v6Searcher = createSearcher(v6Version, v6Content, v6DbPath, v6CachePolicy)
|
|
||||||
println("done")
|
|
||||||
|
|
||||||
println("ip2region search service test program")
|
|
||||||
println("type 'quit' to exit")
|
|
||||||
|
|
||||||
let reader = getStdIn()
|
|
||||||
while (true) {
|
|
||||||
print("ip2region>> ")
|
|
||||||
let lineOpt = reader.readln()
|
|
||||||
if (lineOpt == None) {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
let line = lineOpt.getOrThrow()
|
|
||||||
if (line == "quit" || line == "exit") {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
if (line.size == 0) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
let start = MonoTime.now()
|
|
||||||
let region = searchIP(line, v4Searcher, v6Searcher)
|
|
||||||
let elapsed = MonoTime.now() - start
|
|
||||||
println("${line} -> ${region} (took: ${elapsed.toMicroseconds()} us)")
|
|
||||||
}
|
|
||||||
|
|
||||||
v4Searcher.close()
|
|
||||||
v6Searcher.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
func runBench(args: Array<String>) {
|
|
||||||
var dbFile = ""
|
|
||||||
var srcFile = ""
|
|
||||||
var cachePolicy = "content"
|
|
||||||
|
|
||||||
var i = 1 // skip command name
|
|
||||||
while (i < args.size) {
|
|
||||||
let arg = args[i]
|
|
||||||
if (arg == "--db" && i + 1 < args.size) {
|
|
||||||
dbFile = args[i + 1]
|
|
||||||
i = i + 2
|
|
||||||
} else if (arg == "--src" && i + 1 < args.size) {
|
|
||||||
srcFile = args[i + 1]
|
|
||||||
i = i + 2
|
|
||||||
} else if (arg == "--cache-policy" && i + 1 < args.size) {
|
|
||||||
cachePolicy = args[i + 1]
|
|
||||||
i = i + 2
|
|
||||||
} else {
|
|
||||||
i = i + 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (dbFile == "" || srcFile == "") {
|
|
||||||
println("ip2region bench [command options]")
|
|
||||||
println("options:")
|
|
||||||
println(" --db <path> ip2region binary xdb file path")
|
|
||||||
println(" --src <path> source ip text file path")
|
|
||||||
println(" --cache-policy <mode> cache policy: file/vectorIndex/content")
|
|
||||||
return
|
|
||||||
}
|
|
||||||
|
|
||||||
println("Loading xdb from: ${dbFile}")
|
|
||||||
let content = File.readFrom(Path(dbFile))
|
|
||||||
let header = newHeaderFromBytes(content)
|
|
||||||
let version = versionFromHeader(header)
|
|
||||||
let searcher = createSearcher(version, content, dbFile, cachePolicy)
|
|
||||||
|
|
||||||
println("Loading source data from: ${srcFile}")
|
|
||||||
let srcContent = File.readFrom(Path(srcFile))
|
|
||||||
let srcStr = String.fromUtf8(srcContent)
|
|
||||||
let lines = srcStr.split("\n")
|
|
||||||
|
|
||||||
var count: Int64 = 0
|
|
||||||
var totalCost: Int64 = 0
|
|
||||||
let tStart = MonoTime.now()
|
|
||||||
|
|
||||||
for (line in lines) {
|
|
||||||
let trimmed = line.trimAsciiStart().trimAsciiEnd()
|
|
||||||
if (trimmed.size == 0) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
let parts = trimmed.split("|")
|
|
||||||
if (parts.size < 3) {
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
|
|
||||||
let ipBytes = parseIP(parts[0])
|
|
||||||
let t0 = MonoTime.now()
|
|
||||||
let _ = searcher.search(ipBytes)
|
|
||||||
let t1 = MonoTime.now()
|
|
||||||
totalCost = totalCost + (t1 - t0).toNanoseconds()
|
|
||||||
count = count + 1
|
|
||||||
|
|
||||||
if (count % 10000 == 0) {
|
|
||||||
print(".")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
let tEnd = MonoTime.now()
|
|
||||||
let totalTime = tEnd - tStart
|
|
||||||
|
|
||||||
println("")
|
|
||||||
println("Bench finished:")
|
|
||||||
println(" cachePolicy: ${cachePolicy}")
|
|
||||||
println(" total: ${count}")
|
|
||||||
println(" took: ${totalTime.toMilliseconds()} ms")
|
|
||||||
if (count > 0) {
|
|
||||||
println(" avg: ${totalCost / count / 1000} us/op")
|
|
||||||
}
|
|
||||||
|
|
||||||
searcher.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
func createSearcher(version: Version, content: Array<Byte>, dbPath: String, cachePolicy: String): Searcher {
|
|
||||||
if (cachePolicy == "content") {
|
|
||||||
println("Using content buffer mode")
|
|
||||||
return Searcher(version, content, dbPath)
|
|
||||||
} else if (cachePolicy == "vectorIndex") {
|
|
||||||
println("Using vector index mode")
|
|
||||||
let vIndex = loadVectorIndex(content)
|
|
||||||
return Searcher(version, dbPath, vIndex)
|
|
||||||
} else {
|
|
||||||
println("Using file only mode")
|
|
||||||
return Searcher(version, dbPath)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func searchIP(ipStr: String, v4Searcher: Searcher, v6Searcher: Searcher): String {
|
|
||||||
let ipBytes = parseIP(ipStr)
|
|
||||||
if (ipBytes.size == 4) {
|
|
||||||
return v4Searcher.search(ipBytes)
|
|
||||||
} else if (ipBytes.size == 16) {
|
|
||||||
return v6Searcher.search(ipBytes)
|
|
||||||
}
|
|
||||||
return "Invalid IP"
|
|
||||||
}
|
|
||||||
|
|
||||||
main(args: Array<String>): Int64 {
|
|
||||||
if (args.size == 0) {
|
|
||||||
printHelp()
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
let cmd = args[0]
|
|
||||||
if (cmd == "search") {
|
|
||||||
runSearch(args)
|
|
||||||
} else if (cmd == "bench") {
|
|
||||||
runBench(args)
|
|
||||||
} else {
|
|
||||||
printHelp()
|
|
||||||
}
|
|
||||||
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
@ -1,72 +0,0 @@
|
||||||
package ip2region.service
|
|
||||||
|
|
||||||
import std.fs.*
|
|
||||||
import ip2region.xdb.*
|
|
||||||
|
|
||||||
// Cache policy constants - match Searcher modes
|
|
||||||
public let FileOnly: Int64 = 0
|
|
||||||
public let VectorIndex: Int64 = 1
|
|
||||||
public let ContentBuff: Int64 = 2
|
|
||||||
|
|
||||||
// Config holds xdb file configuration and pre-loaded data for creating Searcher instances
|
|
||||||
public class Config {
|
|
||||||
public let cachePolicy: Int64
|
|
||||||
public let ipVersion: Version
|
|
||||||
public let xdbPath: String
|
|
||||||
public let header: Header
|
|
||||||
public let vIndex: Array<Byte>
|
|
||||||
public let cBuffer: Array<Byte>
|
|
||||||
public let searchers: Int64
|
|
||||||
|
|
||||||
public init(cachePolicy: Int64, ipVersion: Version, xdbPath: String, searchers: Int64) {
|
|
||||||
if (searchers <= 0) {
|
|
||||||
throw Exception("searchers must be > 0")
|
|
||||||
}
|
|
||||||
|
|
||||||
this.cachePolicy = cachePolicy
|
|
||||||
this.ipVersion = ipVersion
|
|
||||||
this.xdbPath = xdbPath
|
|
||||||
this.searchers = searchers
|
|
||||||
|
|
||||||
let content = File.readFrom(Path(xdbPath))
|
|
||||||
this.header = newHeaderFromBytes(content)
|
|
||||||
|
|
||||||
// Verify IP version matches
|
|
||||||
let detectedVersion = versionFromHeader(this.header)
|
|
||||||
if (detectedVersion.id != ipVersion.id) {
|
|
||||||
throw Exception("xdb file IP version mismatch: expected ${ipVersion.name}, got ${detectedVersion.name}")
|
|
||||||
}
|
|
||||||
|
|
||||||
if (cachePolicy == VectorIndex) {
|
|
||||||
this.vIndex = loadVectorIndex(content)
|
|
||||||
this.cBuffer = Array<Byte>(0, repeat: 0)
|
|
||||||
} else if (cachePolicy == ContentBuff) {
|
|
||||||
this.vIndex = Array<Byte>(0, repeat: 0)
|
|
||||||
this.cBuffer = content
|
|
||||||
} else {
|
|
||||||
this.vIndex = Array<Byte>(0, repeat: 0)
|
|
||||||
this.cBuffer = Array<Byte>(0, repeat: 0)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
public init(cachePolicy: Int64, ipVersion: Version, xdbPath: String) {
|
|
||||||
this(cachePolicy, ipVersion, xdbPath, 20)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Static factories for IPv4 and IPv6
|
|
||||||
public func newV4Config(cachePolicy: Int64, xdbPath: String, searchers: Int64): Config {
|
|
||||||
return Config(cachePolicy, IPv4, xdbPath, searchers)
|
|
||||||
}
|
|
||||||
|
|
||||||
public func newV6Config(cachePolicy: Int64, xdbPath: String, searchers: Int64): Config {
|
|
||||||
return Config(cachePolicy, IPv6, xdbPath, searchers)
|
|
||||||
}
|
|
||||||
|
|
||||||
public func newV4Config(cachePolicy: Int64, xdbPath: String): Config {
|
|
||||||
return Config(cachePolicy, IPv4, xdbPath, 20)
|
|
||||||
}
|
|
||||||
|
|
||||||
public func newV6Config(cachePolicy: Int64, xdbPath: String): Config {
|
|
||||||
return Config(cachePolicy, IPv6, xdbPath, 20)
|
|
||||||
}
|
|
||||||
|
|
@ -1,161 +0,0 @@
|
||||||
package ip2region.service
|
|
||||||
|
|
||||||
import ip2region.xdb.*
|
|
||||||
|
|
||||||
// Ip2Region is the high-level API for searching IP addresses.
|
|
||||||
// It supports both IPv4 and IPv6 with thread-safe access via SearcherPool.
|
|
||||||
public class Ip2Region {
|
|
||||||
let v4Pool: SearcherPool
|
|
||||||
let v6Pool: SearcherPool
|
|
||||||
let v4InMemSearcher: Searcher
|
|
||||||
let v6InMemSearcher: Searcher
|
|
||||||
let hasV4: Bool
|
|
||||||
let hasV6: Bool
|
|
||||||
let v4IsInMem: Bool
|
|
||||||
let v6IsInMem: Bool
|
|
||||||
|
|
||||||
// Create Ip2Region with both v4 and v6 configurations.
|
|
||||||
public init(v4Config: Config, v6Config: Config) {
|
|
||||||
// v4 setup
|
|
||||||
if (v4Config.cachePolicy == ContentBuff) {
|
|
||||||
this.hasV4 = true
|
|
||||||
this.v4IsInMem = true
|
|
||||||
this.v4InMemSearcher = Searcher(v4Config.ipVersion, v4Config.cBuffer, v4Config.xdbPath)
|
|
||||||
this.v4Pool = SearcherPool(v4Config)
|
|
||||||
} else {
|
|
||||||
this.hasV4 = true
|
|
||||||
this.v4IsInMem = false
|
|
||||||
this.v4InMemSearcher = Searcher(v4Config.ipVersion, Array<Byte>(0, repeat: 0), v4Config.xdbPath)
|
|
||||||
this.v4Pool = SearcherPool(v4Config)
|
|
||||||
}
|
|
||||||
|
|
||||||
// v6 setup
|
|
||||||
if (v6Config.cachePolicy == ContentBuff) {
|
|
||||||
this.hasV6 = true
|
|
||||||
this.v6IsInMem = true
|
|
||||||
this.v6InMemSearcher = Searcher(v6Config.ipVersion, v6Config.cBuffer, v6Config.xdbPath)
|
|
||||||
this.v6Pool = SearcherPool(v6Config)
|
|
||||||
} else {
|
|
||||||
this.hasV6 = true
|
|
||||||
this.v6IsInMem = false
|
|
||||||
this.v6InMemSearcher = Searcher(v6Config.ipVersion, Array<Byte>(0, repeat: 0), v6Config.xdbPath)
|
|
||||||
this.v6Pool = SearcherPool(v6Config)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Single-version: provide Config and IP version.
|
|
||||||
// Examples:
|
|
||||||
// Ip2Region(v4Config, IPv4) — IPv4 only
|
|
||||||
// Ip2Region(v6Config, IPv6) — IPv6 only
|
|
||||||
public init(config: Config, ipVersion: Version) {
|
|
||||||
if (ipVersion.id == IPv4VersionNo) {
|
|
||||||
this.hasV4 = true
|
|
||||||
this.v4IsInMem = (config.cachePolicy == ContentBuff)
|
|
||||||
if (config.cachePolicy == ContentBuff) {
|
|
||||||
this.v4InMemSearcher = Searcher(config.ipVersion, config.cBuffer, config.xdbPath)
|
|
||||||
} else {
|
|
||||||
this.v4InMemSearcher = Searcher(config.ipVersion, Array<Byte>(0, repeat: 0), config.xdbPath)
|
|
||||||
}
|
|
||||||
this.v4Pool = SearcherPool(config)
|
|
||||||
// IPv6 disabled
|
|
||||||
this.hasV6 = false
|
|
||||||
this.v6IsInMem = false
|
|
||||||
this.v6InMemSearcher = Searcher(IPv4, Array<Byte>(0, repeat: 0), config.xdbPath)
|
|
||||||
this.v6Pool = SearcherPool(Config(ContentBuff, IPv4, config.xdbPath, 1))
|
|
||||||
} else {
|
|
||||||
this.hasV6 = true
|
|
||||||
this.v6IsInMem = (config.cachePolicy == ContentBuff)
|
|
||||||
if (config.cachePolicy == ContentBuff) {
|
|
||||||
this.v6InMemSearcher = Searcher(config.ipVersion, config.cBuffer, config.xdbPath)
|
|
||||||
} else {
|
|
||||||
this.v6InMemSearcher = Searcher(config.ipVersion, Array<Byte>(0, repeat: 0), config.xdbPath)
|
|
||||||
}
|
|
||||||
this.v6Pool = SearcherPool(config)
|
|
||||||
// IPv4 disabled
|
|
||||||
this.hasV4 = false
|
|
||||||
this.v4IsInMem = false
|
|
||||||
this.v4InMemSearcher = Searcher(IPv6, Array<Byte>(0, repeat: 0), config.xdbPath)
|
|
||||||
this.v4Pool = SearcherPool(Config(ContentBuff, IPv6, config.xdbPath, 1))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Search an IP string and return the region info.
|
|
||||||
public func search(ipStr: String): String {
|
|
||||||
try {
|
|
||||||
let ipBytes = parseIP(ipStr)
|
|
||||||
|
|
||||||
if (ipBytes.size == 4) {
|
|
||||||
return this.searchV4(ipBytes)
|
|
||||||
} else if (ipBytes.size == 16) {
|
|
||||||
return this.searchV6(ipBytes)
|
|
||||||
}
|
|
||||||
} catch (_) {
|
|
||||||
return "Invalid IP"
|
|
||||||
}
|
|
||||||
return "Invalid IP"
|
|
||||||
}
|
|
||||||
|
|
||||||
func searchV4(ip: Array<Byte>): String {
|
|
||||||
if (!this.hasV4) {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
if (this.v4IsInMem) {
|
|
||||||
return this.v4InMemSearcher.search(ip)
|
|
||||||
}
|
|
||||||
let searcher = this.v4Pool.borrow()
|
|
||||||
let region = searcher.search(ip)
|
|
||||||
this.v4Pool.return_(searcher)
|
|
||||||
return region
|
|
||||||
}
|
|
||||||
|
|
||||||
func searchV6(ip: Array<Byte>): String {
|
|
||||||
if (!this.hasV6) {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
if (this.v6IsInMem) {
|
|
||||||
return this.v6InMemSearcher.search(ip)
|
|
||||||
}
|
|
||||||
let searcher = this.v6Pool.borrow()
|
|
||||||
let region = searcher.search(ip)
|
|
||||||
this.v6Pool.return_(searcher)
|
|
||||||
return region
|
|
||||||
}
|
|
||||||
|
|
||||||
// close releases all resources.
|
|
||||||
public func close() {
|
|
||||||
if (this.hasV4) {
|
|
||||||
this.v4Pool.close()
|
|
||||||
}
|
|
||||||
if (this.hasV6) {
|
|
||||||
this.v6Pool.close()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// closeTimeout releases all resources with the specified timeout.
|
|
||||||
public func closeTimeout(timeout: Duration) {
|
|
||||||
if (this.hasV4) {
|
|
||||||
this.v4Pool.closeTimeout(timeout)
|
|
||||||
}
|
|
||||||
if (this.hasV6) {
|
|
||||||
this.v6Pool.closeTimeout(timeout)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Create an Ip2Region instance using xdb file paths.
|
|
||||||
// Uses VectorIndex cache policy with 20 searchers by default.
|
|
||||||
public func newIp2Region(v4XdbPath: String, v6XdbPath: String): Ip2Region {
|
|
||||||
let v4Config = Config(VectorIndex, IPv4, v4XdbPath, 20)
|
|
||||||
let v6Config = Config(VectorIndex, IPv6, v6XdbPath, 20)
|
|
||||||
return Ip2Region(v4Config, v6Config)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Create an IPv4-only Ip2Region service.
|
|
||||||
public func newIp2RegionV4(config: Config): Ip2Region {
|
|
||||||
return Ip2Region(config, IPv4)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Create an IPv6-only Ip2Region service.
|
|
||||||
public func newIp2RegionV6(config: Config): Ip2Region {
|
|
||||||
return Ip2Region(config, IPv6)
|
|
||||||
}
|
|
||||||
|
|
@ -1,111 +0,0 @@
|
||||||
package ip2region.service
|
|
||||||
|
|
||||||
import std.sync.*
|
|
||||||
import std.core.*
|
|
||||||
import std.time.*
|
|
||||||
import std.collection.concurrent.*
|
|
||||||
import ip2region.xdb.*
|
|
||||||
|
|
||||||
// SearcherPool is a thread-safe pool of xdb.Searcher instances.
|
|
||||||
// Uses Semaphore for backpressure and ConcurrentLinkedQueue for storage.
|
|
||||||
public class SearcherPool {
|
|
||||||
let config: Config
|
|
||||||
let queue: ConcurrentLinkedQueue<Searcher>
|
|
||||||
let semaphore: Semaphore
|
|
||||||
let closing: AtomicBool
|
|
||||||
let loanCnt: AtomicInt64
|
|
||||||
|
|
||||||
public init(config: Config) {
|
|
||||||
this.config = config
|
|
||||||
this.queue = ConcurrentLinkedQueue<Searcher>()
|
|
||||||
this.semaphore = Semaphore(config.searchers)
|
|
||||||
this.closing = AtomicBool(false)
|
|
||||||
this.loanCnt = AtomicInt64(0)
|
|
||||||
|
|
||||||
let poolSize = config.searchers
|
|
||||||
for (_ in 0..poolSize) {
|
|
||||||
let searcher = createPoolSearcher(config)
|
|
||||||
this.queue.add(searcher)
|
|
||||||
this.semaphore.release(amount: 1)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// borrow takes a Searcher from the pool, blocking until one is available.
|
|
||||||
// Throws if the pool is closing or closed.
|
|
||||||
public func borrow(): Searcher {
|
|
||||||
while (!this.closing.load()) {
|
|
||||||
if (this.semaphore.tryAcquire(amount: 1)) {
|
|
||||||
let opt = this.queue.remove()
|
|
||||||
if (opt.isNone()) {
|
|
||||||
this.semaphore.release(amount: 1)
|
|
||||||
continue
|
|
||||||
}
|
|
||||||
let s = opt.getOrThrow()
|
|
||||||
this.loanCnt.fetchAdd(1)
|
|
||||||
return s
|
|
||||||
}
|
|
||||||
sleep(Duration.millisecond * 5)
|
|
||||||
}
|
|
||||||
throw Exception("SearcherPool is closing")
|
|
||||||
}
|
|
||||||
|
|
||||||
// return_ returns a Searcher to the pool, or closes it if the pool is shutting down.
|
|
||||||
public func return_(searcher: Searcher) {
|
|
||||||
if (this.closing.load()) {
|
|
||||||
searcher.close()
|
|
||||||
} else {
|
|
||||||
this.queue.add(searcher)
|
|
||||||
this.semaphore.release(amount: 1)
|
|
||||||
}
|
|
||||||
this.loanCnt.fetchSub(1)
|
|
||||||
}
|
|
||||||
|
|
||||||
// close closes the pool with a default 10 second timeout.
|
|
||||||
public func close() {
|
|
||||||
this.closeTimeout(Duration.second * 10)
|
|
||||||
}
|
|
||||||
|
|
||||||
// closeTimeout closes the pool with the specified timeout.
|
|
||||||
public func closeTimeout(timeout: Duration) {
|
|
||||||
this.closing.store(true)
|
|
||||||
|
|
||||||
// Drain the queue: acquire all permits and close searchers
|
|
||||||
while (true) {
|
|
||||||
if (!this.semaphore.tryAcquire(amount: 1)) {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
try {
|
|
||||||
let opt = this.queue.remove()
|
|
||||||
if (opt.isSome()) {
|
|
||||||
opt.getOrThrow().close()
|
|
||||||
}
|
|
||||||
} catch (_) {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Wait for outstanding loans to return (with timeout)
|
|
||||||
let deadline = MonoTime.now() + timeout
|
|
||||||
while (this.loanCnt.load() > 0) {
|
|
||||||
if (MonoTime.now() >= deadline) {
|
|
||||||
break
|
|
||||||
}
|
|
||||||
sleep(Duration.millisecond * 10)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
public func getLoanCount(): Int64 {
|
|
||||||
return this.loanCnt.load()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Helper to create a Searcher based on Config's cache policy
|
|
||||||
func createPoolSearcher(config: Config): Searcher {
|
|
||||||
if (config.cachePolicy == ContentBuff) {
|
|
||||||
return Searcher(config.ipVersion, config.cBuffer, config.xdbPath)
|
|
||||||
} else if (config.cachePolicy == VectorIndex) {
|
|
||||||
return Searcher(config.ipVersion, config.xdbPath, config.vIndex)
|
|
||||||
} else {
|
|
||||||
return Searcher(config.ipVersion, config.xdbPath)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,50 +0,0 @@
|
||||||
package ip2region.tests.service
|
|
||||||
|
|
||||||
import ip2region.xdb.*
|
|
||||||
import ip2region.service.*
|
|
||||||
|
|
||||||
@Test
|
|
||||||
class ConfigV4Tests {
|
|
||||||
@TestCase
|
|
||||||
func testCreateFileOnly() {
|
|
||||||
let cfg = Config(FileOnly, IPv4, "../../data/ip2region_v4.xdb", 5)
|
|
||||||
if (cfg.cachePolicy != FileOnly) { throw Exception("wrong cache policy") }
|
|
||||||
if (cfg.searchers != 5) { throw Exception("wrong searchers count") }
|
|
||||||
if (cfg.ipVersion.id != IPv4VersionNo) { throw Exception("wrong version") }
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testCreateVectorIndex() {
|
|
||||||
let cfg = Config(VectorIndex, IPv4, "../../data/ip2region_v4.xdb", 10)
|
|
||||||
if (cfg.vIndex.size == 0) { throw Exception("vIndex should be loaded") }
|
|
||||||
let expectedSize = VectorIndexRows * VectorIndexCols * VectorIndexSize
|
|
||||||
if (cfg.vIndex.size != expectedSize) {
|
|
||||||
throw Exception("vIndex size mismatch: ${cfg.vIndex.size} vs ${expectedSize}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testCreateContentBuff() {
|
|
||||||
let cfg = Config(ContentBuff, IPv4, "../../data/ip2region_v4.xdb", 3)
|
|
||||||
if (cfg.cBuffer.size == 0) { throw Exception("cBuffer should be loaded") }
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testFactoryV4() {
|
|
||||||
let cfg = newV4Config(VectorIndex, "../../data/ip2region_v4.xdb", 8)
|
|
||||||
if (cfg.ipVersion.id != IPv4VersionNo) { throw Exception("should be IPv4") }
|
|
||||||
if (cfg.searchers != 8) { throw Exception("wrong searchers") }
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testFactoryV6() {
|
|
||||||
let cfg = newV6Config(VectorIndex, "../../data/ip2region_v6.xdb", 8)
|
|
||||||
if (cfg.ipVersion.id != IPv6VersionNo) { throw Exception("should be IPv6") }
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testFactoryDefaultSearchers() {
|
|
||||||
let cfg = newV4Config(ContentBuff, "../../data/ip2region_v4.xdb")
|
|
||||||
if (cfg.searchers != 20) { throw Exception("default searchers should be 20") }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,108 +0,0 @@
|
||||||
package ip2region.tests.service
|
|
||||||
|
|
||||||
import ip2region.xdb.*
|
|
||||||
import ip2region.service.*
|
|
||||||
|
|
||||||
@Test
|
|
||||||
class Ip2RegionBasicTests {
|
|
||||||
@TestCase
|
|
||||||
func testSearchWithContentBuff() {
|
|
||||||
let v4Cfg = Config(ContentBuff, IPv4, "../../data/ip2region_v4.xdb", 1)
|
|
||||||
let v6Cfg = Config(ContentBuff, IPv6, "../../data/ip2region_v6.xdb", 1)
|
|
||||||
let region = Ip2Region(v4Cfg, v6Cfg)
|
|
||||||
let r1 = region.search("220.181.108.183")
|
|
||||||
if (r1.size == 0) {
|
|
||||||
throw Exception("v4 search should return result")
|
|
||||||
}
|
|
||||||
let r2 = region.search("2408:8266:100:1000::")
|
|
||||||
if (r2.size == 0) {
|
|
||||||
throw Exception("v6 search should return result")
|
|
||||||
}
|
|
||||||
region.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testSearchWithVectorIndex() {
|
|
||||||
let v4Cfg = Config(VectorIndex, IPv4, "../../data/ip2region_v4.xdb", 5)
|
|
||||||
let v6Cfg = Config(VectorIndex, IPv6, "../../data/ip2region_v6.xdb", 5)
|
|
||||||
let region = Ip2Region(v4Cfg, v6Cfg)
|
|
||||||
let r = region.search("220.181.108.183")
|
|
||||||
if (r.size == 0) {
|
|
||||||
throw Exception("v4 search should return result")
|
|
||||||
}
|
|
||||||
region.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testInvalidIP() {
|
|
||||||
let v4Cfg = Config(ContentBuff, IPv4, "../../data/ip2region_v4.xdb", 1)
|
|
||||||
let v6Cfg = Config(ContentBuff, IPv6, "../../data/ip2region_v6.xdb", 1)
|
|
||||||
let region = Ip2Region(v4Cfg, v6Cfg)
|
|
||||||
let r = region.search("invalid")
|
|
||||||
if (r != "Invalid IP") {
|
|
||||||
throw Exception("expected 'Invalid IP', got '${r}'")
|
|
||||||
}
|
|
||||||
region.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testClose() {
|
|
||||||
let v4Cfg = Config(ContentBuff, IPv4, "../../data/ip2region_v4.xdb", 1)
|
|
||||||
let v6Cfg = Config(ContentBuff, IPv6, "../../data/ip2region_v6.xdb", 1)
|
|
||||||
let region = Ip2Region(v4Cfg, v6Cfg)
|
|
||||||
region.search("220.181.108.183")
|
|
||||||
region.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testV4OnlyInit() {
|
|
||||||
let cfg = Config(VectorIndex, IPv4, "../../data/ip2region_v4.xdb", 3)
|
|
||||||
let region = Ip2Region(cfg, IPv4)
|
|
||||||
let r = region.search("220.181.108.183")
|
|
||||||
if (r.size == 0) {
|
|
||||||
throw Exception("v4-only region should find v4 IP")
|
|
||||||
}
|
|
||||||
let r6 = region.search("2408:8266:100:1000::")
|
|
||||||
if (r6.size != 0) {
|
|
||||||
throw Exception("v4-only region should not find v6 IP")
|
|
||||||
}
|
|
||||||
region.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testV6OnlyInit() {
|
|
||||||
let cfg = Config(VectorIndex, IPv6, "../../data/ip2region_v6.xdb", 3)
|
|
||||||
let region = Ip2Region(cfg, IPv6)
|
|
||||||
let r = region.search("2408:8266:100:1000::")
|
|
||||||
if (r.size == 0) {
|
|
||||||
throw Exception("v6-only region should find v6 IP")
|
|
||||||
}
|
|
||||||
let r4 = region.search("220.181.108.183")
|
|
||||||
if (r4.size != 0) {
|
|
||||||
throw Exception("v6-only region should not find v4 IP")
|
|
||||||
}
|
|
||||||
region.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testNewV4Factory() {
|
|
||||||
let cfg = Config(ContentBuff, IPv4, "../../data/ip2region_v4.xdb", 2)
|
|
||||||
let region = newIp2RegionV4(cfg)
|
|
||||||
let r = region.search("220.181.108.183")
|
|
||||||
if (r.size == 0) {
|
|
||||||
throw Exception("newIp2RegionV4 should work")
|
|
||||||
}
|
|
||||||
region.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testNewV6Factory() {
|
|
||||||
let cfg = Config(ContentBuff, IPv6, "../../data/ip2region_v6.xdb", 2)
|
|
||||||
let region = newIp2RegionV6(cfg)
|
|
||||||
let r = region.search("2408:8266:100:1000::")
|
|
||||||
if (r.size == 0) {
|
|
||||||
throw Exception("newIp2RegionV6 should work")
|
|
||||||
}
|
|
||||||
region.close()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,55 +0,0 @@
|
||||||
package ip2region.tests.service
|
|
||||||
|
|
||||||
import ip2region.xdb.*
|
|
||||||
import ip2region.service.*
|
|
||||||
|
|
||||||
@Test
|
|
||||||
class SearcherPoolTests {
|
|
||||||
@TestCase
|
|
||||||
func testBorrowAndReturn() {
|
|
||||||
let cfg = Config(ContentBuff, IPv4, "../../data/ip2region_v4.xdb", 3)
|
|
||||||
let pool = SearcherPool(cfg)
|
|
||||||
let s = pool.borrow()
|
|
||||||
if (pool.getLoanCount() != 1) {
|
|
||||||
throw Exception("loan count should be 1, got ${pool.getLoanCount()}")
|
|
||||||
}
|
|
||||||
pool.return_(s)
|
|
||||||
if (pool.getLoanCount() != 0) {
|
|
||||||
throw Exception("loan count should be 0 after return")
|
|
||||||
}
|
|
||||||
pool.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testBorrowAll() {
|
|
||||||
let cfg = Config(ContentBuff, IPv4, "../../data/ip2region_v4.xdb", 5)
|
|
||||||
let pool = SearcherPool(cfg)
|
|
||||||
let s1 = pool.borrow()
|
|
||||||
let s2 = pool.borrow()
|
|
||||||
let s3 = pool.borrow()
|
|
||||||
let s4 = pool.borrow()
|
|
||||||
let s5 = pool.borrow()
|
|
||||||
if (pool.getLoanCount() != 5) {
|
|
||||||
throw Exception("should have 5 loans, got ${pool.getLoanCount()}")
|
|
||||||
}
|
|
||||||
pool.return_(s5)
|
|
||||||
pool.return_(s4)
|
|
||||||
pool.return_(s3)
|
|
||||||
pool.return_(s2)
|
|
||||||
pool.return_(s1)
|
|
||||||
if (pool.getLoanCount() != 0) {
|
|
||||||
throw Exception("all should be returned, got ${pool.getLoanCount()}")
|
|
||||||
}
|
|
||||||
pool.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testClosePool() {
|
|
||||||
let cfg = Config(ContentBuff, IPv4, "../../data/ip2region_v4.xdb", 2)
|
|
||||||
let pool = SearcherPool(cfg)
|
|
||||||
let s = pool.borrow()
|
|
||||||
pool.return_(s)
|
|
||||||
pool.close()
|
|
||||||
pool.close()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
package ip2region.tests
|
|
||||||
|
|
@ -1,83 +0,0 @@
|
||||||
package ip2region.tests.xdb
|
|
||||||
|
|
||||||
import ip2region.xdb.*
|
|
||||||
|
|
||||||
@Test
|
|
||||||
class HeaderTests {
|
|
||||||
func makeHeaderBytes(): Array<Byte> {
|
|
||||||
let buf = Array<Byte>(256, repeat: 0)
|
|
||||||
var v: Byte
|
|
||||||
v = 3; buf[0] = v; v = 0; buf[1] = v
|
|
||||||
v = 1; buf[2] = v; v = 0; buf[3] = v
|
|
||||||
v = 0xA0; buf[4] = v; v = 0x86; buf[5] = v; v = 0x01; buf[6] = v; v = 0x00; buf[7] = v
|
|
||||||
v = 0xE8; buf[8] = v; v = 0x03; buf[9] = v; v = 0x00; buf[10] = v; v = 0x00; buf[11] = v
|
|
||||||
v = 0xD0; buf[12] = v; v = 0x07; buf[13] = v; v = 0x00; buf[14] = v; v = 0x00; buf[15] = v
|
|
||||||
v = 4; buf[16] = v; v = 0; buf[17] = v
|
|
||||||
v = 0; buf[18] = v; v = 0; buf[19] = v
|
|
||||||
return buf
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testNewHeaderFromBytes() {
|
|
||||||
let buf = makeHeaderBytes()
|
|
||||||
let h = newHeaderFromBytes(buf)
|
|
||||||
if (h.version != UInt16(3)) {
|
|
||||||
throw Exception("expected version 3, got ${h.version}")
|
|
||||||
}
|
|
||||||
if (h.ipVersion != UInt16(4)) {
|
|
||||||
throw Exception("expected ipVersion 4, got ${h.ipVersion}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testHeaderToString() {
|
|
||||||
let buf = makeHeaderBytes()
|
|
||||||
let h = newHeaderFromBytes(buf)
|
|
||||||
let s = h.toString()
|
|
||||||
if (s.size == 0) {
|
|
||||||
throw Exception("toString should not be empty")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testVersionFromHeaderV4() {
|
|
||||||
let buf = makeHeaderBytes()
|
|
||||||
let h = newHeaderFromBytes(buf)
|
|
||||||
let v = versionFromHeader(h)
|
|
||||||
if (v.id != IPv4VersionNo) {
|
|
||||||
throw Exception("expected IPv4, got ${v.name}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testVersionFromHeaderV6() {
|
|
||||||
let buf = makeHeaderBytes()
|
|
||||||
var v: Byte = 6; buf[16] = v
|
|
||||||
let h = newHeaderFromBytes(buf)
|
|
||||||
let ver = versionFromHeader(h)
|
|
||||||
if (ver.id != IPv6VersionNo) {
|
|
||||||
throw Exception("expected IPv6, got ${ver.name}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testVersionFromHeader20() {
|
|
||||||
let buf = makeHeaderBytes()
|
|
||||||
var v: Byte = 2; buf[0] = v
|
|
||||||
v = 0; buf[16] = v
|
|
||||||
let h = newHeaderFromBytes(buf)
|
|
||||||
let ver = versionFromHeader(h)
|
|
||||||
if (ver.id != IPv4VersionNo) {
|
|
||||||
throw Exception("expected IPv4 for v2.0")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testHeaderConstants() {
|
|
||||||
if (HeaderInfoLength != 256) { throw Exception("expected 256") }
|
|
||||||
if (VectorIndexRows != 256) { throw Exception("expected 256") }
|
|
||||||
if (VectorIndexSize != 8) { throw Exception("expected 8") }
|
|
||||||
if (IPv4SegmentIndexSize != 14) { throw Exception("expected 14") }
|
|
||||||
if (IPv6SegmentIndexSize != 38) { throw Exception("expected 38") }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,132 +0,0 @@
|
||||||
package ip2region.tests.xdb
|
|
||||||
|
|
||||||
import std.fs.*
|
|
||||||
import ip2region.xdb.*
|
|
||||||
|
|
||||||
@Test
|
|
||||||
class SearcherV4Tests {
|
|
||||||
var v4Content: Array<Byte> = Array<Byte>(0, repeat: 0)
|
|
||||||
var initialized: Bool = false
|
|
||||||
|
|
||||||
@BeforeAll
|
|
||||||
func setup() {
|
|
||||||
this.v4Content = File.readFrom(Path("../../data/ip2region_v4.xdb"))
|
|
||||||
this.initialized = true
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testSearchContentMode() {
|
|
||||||
if (!this.initialized) { return }
|
|
||||||
let header = newHeaderFromBytes(this.v4Content)
|
|
||||||
let version = versionFromHeader(header)
|
|
||||||
let searcher = Searcher(version, this.v4Content, "../../data/ip2region_v4.xdb")
|
|
||||||
let ip = parseIP("220.181.108.183")
|
|
||||||
let region = searcher.search(ip)
|
|
||||||
if (region.size == 0) {
|
|
||||||
throw Exception("should find region for known IP")
|
|
||||||
}
|
|
||||||
searcher.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testSearchVectorIndexMode() {
|
|
||||||
if (!this.initialized) { return }
|
|
||||||
let header = newHeaderFromBytes(this.v4Content)
|
|
||||||
let version = versionFromHeader(header)
|
|
||||||
let vIndex = loadVectorIndex(this.v4Content)
|
|
||||||
let searcher = Searcher(version, "../../data/ip2region_v4.xdb", vIndex)
|
|
||||||
let ip = parseIP("220.181.108.183")
|
|
||||||
let region = searcher.search(ip)
|
|
||||||
if (region.size == 0) {
|
|
||||||
throw Exception("should find region for known IP")
|
|
||||||
}
|
|
||||||
searcher.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testSearchFileMode() {
|
|
||||||
if (!this.initialized) { return }
|
|
||||||
let header = newHeaderFromBytes(this.v4Content)
|
|
||||||
let version = versionFromHeader(header)
|
|
||||||
let searcher = Searcher(version, "../../data/ip2region_v4.xdb")
|
|
||||||
let ip = parseIP("220.181.108.183")
|
|
||||||
let region = searcher.search(ip)
|
|
||||||
if (region.size == 0) {
|
|
||||||
throw Exception("should find region for known IP")
|
|
||||||
}
|
|
||||||
searcher.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testAllModesSameResult() {
|
|
||||||
if (!this.initialized) { return }
|
|
||||||
let header = newHeaderFromBytes(this.v4Content)
|
|
||||||
let version = versionFromHeader(header)
|
|
||||||
let vIndex = loadVectorIndex(this.v4Content)
|
|
||||||
let s1 = Searcher(version, this.v4Content, "../../data/ip2region_v4.xdb")
|
|
||||||
let s2 = Searcher(version, "../../data/ip2region_v4.xdb", vIndex)
|
|
||||||
let s3 = Searcher(version, "../../data/ip2region_v4.xdb")
|
|
||||||
let ip = parseIP("1.2.3.4")
|
|
||||||
let r1 = s1.search(ip)
|
|
||||||
let r2 = s2.search(ip)
|
|
||||||
let r3 = s3.search(ip)
|
|
||||||
s1.close(); s2.close(); s3.close()
|
|
||||||
if (r1 != r2 || r2 != r3) {
|
|
||||||
throw Exception("all modes should return same result")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testSearchByString() {
|
|
||||||
if (!this.initialized) { return }
|
|
||||||
let header = newHeaderFromBytes(this.v4Content)
|
|
||||||
let version = versionFromHeader(header)
|
|
||||||
let searcher = Searcher(version, this.v4Content, "../../data/ip2region_v4.xdb")
|
|
||||||
let region = searcher.searchByString("220.181.108.183")
|
|
||||||
if (region.size == 0) {
|
|
||||||
throw Exception("searchByString should find region")
|
|
||||||
}
|
|
||||||
searcher.close()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@Test
|
|
||||||
class SearcherV6Tests {
|
|
||||||
var v6Content: Array<Byte> = Array<Byte>(0, repeat: 0)
|
|
||||||
var initialized: Bool = false
|
|
||||||
|
|
||||||
@BeforeAll
|
|
||||||
func setup() {
|
|
||||||
this.v6Content = File.readFrom(Path("../../data/ip2region_v6.xdb"))
|
|
||||||
this.initialized = true
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testSearchV6ContentMode() {
|
|
||||||
if (!this.initialized) { return }
|
|
||||||
let header = newHeaderFromBytes(this.v6Content)
|
|
||||||
let version = versionFromHeader(header)
|
|
||||||
let searcher = Searcher(version, this.v6Content, "../../data/ip2region_v6.xdb")
|
|
||||||
let ip = parseIP("2408:8266:100:1000::")
|
|
||||||
let region = searcher.search(ip)
|
|
||||||
if (region.size == 0) {
|
|
||||||
throw Exception("should find region for known IPv6")
|
|
||||||
}
|
|
||||||
searcher.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testSearchV6VectorIndexMode() {
|
|
||||||
if (!this.initialized) { return }
|
|
||||||
let header = newHeaderFromBytes(this.v6Content)
|
|
||||||
let version = versionFromHeader(header)
|
|
||||||
let vIndex = loadVectorIndex(this.v6Content)
|
|
||||||
let searcher = Searcher(version, "../../data/ip2region_v6.xdb", vIndex)
|
|
||||||
let ip = parseIP("2408:8266:100:1000::")
|
|
||||||
let region = searcher.search(ip)
|
|
||||||
if (region.size == 0) {
|
|
||||||
throw Exception("should find region for known IPv6")
|
|
||||||
}
|
|
||||||
searcher.close()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,133 +0,0 @@
|
||||||
package ip2region.tests.xdb
|
|
||||||
|
|
||||||
import ip2region.xdb.*
|
|
||||||
|
|
||||||
@Test
|
|
||||||
class UtilTests {
|
|
||||||
@TestCase
|
|
||||||
func testParseIPv4() {
|
|
||||||
let ip = parseIP("1.2.3.4")
|
|
||||||
if (ip.size != 4) {
|
|
||||||
throw Exception("expected 4 bytes, got ${ip.size}")
|
|
||||||
}
|
|
||||||
if (Int64(ip[0]) != 1 || Int64(ip[1]) != 2 || Int64(ip[2]) != 3 || Int64(ip[3]) != 4) {
|
|
||||||
throw Exception("IPv4 parse mismatch")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testParseIPv6() {
|
|
||||||
let ip = parseIP("::1")
|
|
||||||
if (ip.size != 16) {
|
|
||||||
throw Exception("expected 16 bytes for IPv6, got ${ip.size}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testIPCompareEqual() {
|
|
||||||
let a = Array<Byte>(4, repeat: 0)
|
|
||||||
var v: Byte = 1; a[0] = v; v = 2; a[1] = v; v = 3; a[2] = v; v = 4; a[3] = v
|
|
||||||
let b = Array<Byte>(4, repeat: 0)
|
|
||||||
v = 1; b[0] = v; v = 2; b[1] = v; v = 3; b[2] = v; v = 4; b[3] = v
|
|
||||||
let r = ipCompare(a, b)
|
|
||||||
if (r != 0) {
|
|
||||||
throw Exception("expected 0, got ${r}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testIPCompareLess() {
|
|
||||||
let a = Array<Byte>(4, repeat: 0)
|
|
||||||
var v: Byte = 1; a[0] = v; v = 2; a[1] = v; v = 3; a[2] = v; v = 4; a[3] = v
|
|
||||||
let b = Array<Byte>(4, repeat: 0)
|
|
||||||
v = 1; b[0] = v; v = 2; b[1] = v; v = 3; b[2] = v; v = 5; b[3] = v
|
|
||||||
let r = ipCompare(a, b)
|
|
||||||
if (r != -1) {
|
|
||||||
throw Exception("expected -1, got ${r}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testIPCompareGreater() {
|
|
||||||
let a = Array<Byte>(4, repeat: 0)
|
|
||||||
var v: Byte = 1; a[0] = v; v = 2; a[1] = v; v = 3; a[2] = v; v = 5; a[3] = v
|
|
||||||
let b = Array<Byte>(4, repeat: 0)
|
|
||||||
v = 1; b[0] = v; v = 2; b[1] = v; v = 3; b[2] = v; v = 4; b[3] = v
|
|
||||||
let r = ipCompare(a, b)
|
|
||||||
if (r != 1) {
|
|
||||||
throw Exception("expected 1, got ${r}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testReadLEUint32() {
|
|
||||||
let buf = Array<Byte>(4, repeat: 0)
|
|
||||||
var v: Byte = 0x78; buf[0] = v; v = 0x56; buf[1] = v; v = 0x34; buf[2] = v; v = 0x12; buf[3] = v
|
|
||||||
let val = readLEUint32(buf, 0)
|
|
||||||
let expected: Int64 = 0x12345678
|
|
||||||
if (val != expected) {
|
|
||||||
throw Exception("expected 0x12345678, got ${val}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testReadLEUint16() {
|
|
||||||
let buf = Array<Byte>(2, repeat: 0)
|
|
||||||
var v: Byte = 0x34; buf[0] = v; v = 0x12; buf[1] = v
|
|
||||||
let val = readLEUint16(buf, 0)
|
|
||||||
if (val != 0x1234) {
|
|
||||||
throw Exception("expected 0x1234, got ${val}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testIpToString() {
|
|
||||||
let ip = Array<Byte>(4, repeat: 0)
|
|
||||||
var v: Byte = 127; ip[0] = v; v = 0; ip[1] = v; v = 0; ip[2] = v; v = 1; ip[3] = v
|
|
||||||
let s = ipToString(ip)
|
|
||||||
if (s != "127.0.0.1") {
|
|
||||||
throw Exception("expected 127.0.0.1, got ${s}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testCompareIPv4() {
|
|
||||||
let ip = Array<Byte>(4, repeat: 0)
|
|
||||||
var v: Byte = 1; ip[0] = v; v = 2; ip[1] = v; v = 3; ip[2] = v; v = 4; ip[3] = v
|
|
||||||
let entry = Array<Byte>(8, repeat: 0)
|
|
||||||
v = 4; entry[0] = v; v = 3; entry[1] = v; v = 2; entry[2] = v; v = 1; entry[3] = v
|
|
||||||
v = 8; entry[4] = v; v = 7; entry[5] = v; v = 6; entry[6] = v; v = 5; entry[7] = v
|
|
||||||
let r = compareIPv4(ip, entry, 0)
|
|
||||||
if (r != 0) {
|
|
||||||
throw Exception("expected equal, got ${r}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testCompareIPv6() {
|
|
||||||
let ip = Array<Byte>(16, repeat: 0)
|
|
||||||
var v: Byte = 0x20; ip[0] = v; v = 0x01; ip[1] = v; v = 0x0d; ip[2] = v; v = 0xb8; ip[3] = v
|
|
||||||
let entry = Array<Byte>(16, repeat: 0)
|
|
||||||
v = 0x20; entry[0] = v; v = 0x01; entry[1] = v; v = 0x0d; entry[2] = v; v = 0xb8; entry[3] = v
|
|
||||||
let r = compareIPv6(ip, entry, 0)
|
|
||||||
if (r != 0) {
|
|
||||||
throw Exception("expected equal, got ${r}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
@TestCase
|
|
||||||
func testLoadVectorIndex() {
|
|
||||||
let viSize = VectorIndexRows * VectorIndexCols * VectorIndexSize
|
|
||||||
let cBuff = Array<Byte>(HeaderInfoLength + viSize, repeat: 0)
|
|
||||||
var v: Byte = 0x42
|
|
||||||
cBuff[HeaderInfoLength] = v
|
|
||||||
let vi = loadVectorIndex(cBuff)
|
|
||||||
if (vi.size != viSize) {
|
|
||||||
throw Exception("expected size ${viSize}, got ${vi.size}")
|
|
||||||
}
|
|
||||||
let firstVal = Int64(vi[0])
|
|
||||||
if (firstVal != 0x42) {
|
|
||||||
throw Exception("expected first byte 0x42, got ${firstVal}")
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,77 +0,0 @@
|
||||||
package ip2region.xdb
|
|
||||||
|
|
||||||
// xdb file format constants
|
|
||||||
public let HeaderInfoLength: Int64 = 256
|
|
||||||
public let VectorIndexRows: Int64 = 256
|
|
||||||
public let VectorIndexCols: Int64 = 256
|
|
||||||
public let VectorIndexSize: Int64 = 8
|
|
||||||
public let IPv4SegmentIndexSize: Int64 = 14 // 4 + 4 + 2 + 4
|
|
||||||
public let IPv6SegmentIndexSize: Int64 = 38 // 16 + 16 + 2 + 4
|
|
||||||
|
|
||||||
public let Structure20: Int64 = 2
|
|
||||||
public let Structure30: Int64 = 3
|
|
||||||
|
|
||||||
// xdb header parsed from the first 256 bytes of the xdb file
|
|
||||||
public class Header {
|
|
||||||
public let version: UInt16
|
|
||||||
public let indexPolicy: UInt16
|
|
||||||
public let createdAt: UInt32
|
|
||||||
public let startIndexPtr: UInt32
|
|
||||||
public let endIndexPtr: UInt32
|
|
||||||
public let ipVersion: UInt16
|
|
||||||
public let runtimePtrBytes: UInt16
|
|
||||||
|
|
||||||
public init(
|
|
||||||
ver: UInt16,
|
|
||||||
idxPolicy: UInt16,
|
|
||||||
createAt: UInt32,
|
|
||||||
sPtr: UInt32,
|
|
||||||
ePtr: UInt32,
|
|
||||||
ipVer: UInt16,
|
|
||||||
rtPtrBytes: UInt16
|
|
||||||
) {
|
|
||||||
this.version = ver
|
|
||||||
this.indexPolicy = idxPolicy
|
|
||||||
this.createdAt = createAt
|
|
||||||
this.startIndexPtr = sPtr
|
|
||||||
this.endIndexPtr = ePtr
|
|
||||||
this.ipVersion = ipVer
|
|
||||||
this.runtimePtrBytes = rtPtrBytes
|
|
||||||
}
|
|
||||||
|
|
||||||
public func toString(): String {
|
|
||||||
return "{version: ${this.version}, index_policy: ${this.indexPolicy}, " +
|
|
||||||
"created_at: ${this.createdAt}, start_index_ptr: ${this.startIndexPtr}, " +
|
|
||||||
"end_index_ptr: ${this.endIndexPtr}, ip_version: ${this.ipVersion}, " +
|
|
||||||
"runtime_ptr_bytes: ${this.runtimePtrBytes}}"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// Read a little-endian UInt16 from byte array at offset
|
|
||||||
func readHeaderUInt16(buf: Array<Byte>, offset: Int64): UInt16 {
|
|
||||||
let b0 = UInt16(buf[offset])
|
|
||||||
let b1 = UInt16(buf[offset + 1]) << 8
|
|
||||||
return b0 | b1
|
|
||||||
}
|
|
||||||
|
|
||||||
// Read a little-endian UInt32 from byte array at offset
|
|
||||||
func readHeaderUInt32(buf: Array<Byte>, offset: Int64): UInt32 {
|
|
||||||
let b0 = UInt32(buf[offset])
|
|
||||||
let b1 = UInt32(buf[offset + 1]) << 8
|
|
||||||
let b2 = UInt32(buf[offset + 2]) << 16
|
|
||||||
let b3 = UInt32(buf[offset + 3]) << 24
|
|
||||||
return b0 | b1 | b2 | b3
|
|
||||||
}
|
|
||||||
|
|
||||||
// Create a Header from raw byte buffer (first 256 bytes of xdb file)
|
|
||||||
public func newHeaderFromBytes(buf: Array<Byte>): Header {
|
|
||||||
return Header(
|
|
||||||
readHeaderUInt16(buf, 0),
|
|
||||||
readHeaderUInt16(buf, 2),
|
|
||||||
readHeaderUInt32(buf, 4),
|
|
||||||
readHeaderUInt32(buf, 8),
|
|
||||||
readHeaderUInt32(buf, 12),
|
|
||||||
readHeaderUInt16(buf, 16),
|
|
||||||
readHeaderUInt16(buf, 18)
|
|
||||||
)
|
|
||||||
}
|
|
||||||
|
|
@ -1,146 +0,0 @@
|
||||||
package ip2region.xdb
|
|
||||||
|
|
||||||
import std.fs.*
|
|
||||||
import std.io.SeekPosition
|
|
||||||
|
|
||||||
// Searcher for ip2region xdb database
|
|
||||||
public class Searcher {
|
|
||||||
let version: Version
|
|
||||||
let handle: File
|
|
||||||
var ioCount: Int64
|
|
||||||
let vectorIndex: Array<Byte>
|
|
||||||
let contentBuff: Array<Byte>
|
|
||||||
let mode: Int64
|
|
||||||
|
|
||||||
public let FileOnlyMode: Int64 = 0
|
|
||||||
public let VectorIndexMode: Int64 = 1
|
|
||||||
public let ContentBuffMode: Int64 = 2
|
|
||||||
|
|
||||||
public init(ver: Version, dbFile: String) {
|
|
||||||
this.version = ver
|
|
||||||
this.handle = File(dbFile, Read)
|
|
||||||
this.vectorIndex = Array<Byte>(0, repeat: 0)
|
|
||||||
this.contentBuff = Array<Byte>(0, repeat: 0)
|
|
||||||
this.mode = FileOnlyMode
|
|
||||||
this.ioCount = 0
|
|
||||||
}
|
|
||||||
|
|
||||||
public init(ver: Version, dbFile: String, vIndex: Array<Byte>) {
|
|
||||||
this.version = ver
|
|
||||||
this.handle = File(dbFile, Read)
|
|
||||||
this.vectorIndex = vIndex
|
|
||||||
this.contentBuff = Array<Byte>(0, repeat: 0)
|
|
||||||
this.mode = VectorIndexMode
|
|
||||||
this.ioCount = 0
|
|
||||||
}
|
|
||||||
|
|
||||||
public init(ver: Version, cBuff: Array<Byte>, dbPath: String) {
|
|
||||||
this.version = ver
|
|
||||||
this.handle = File(dbPath, Read)
|
|
||||||
this.vectorIndex = Array<Byte>(0, repeat: 0)
|
|
||||||
this.contentBuff = cBuff
|
|
||||||
this.mode = ContentBuffMode
|
|
||||||
this.ioCount = 0
|
|
||||||
}
|
|
||||||
|
|
||||||
public func close() {
|
|
||||||
this.handle.close()
|
|
||||||
}
|
|
||||||
|
|
||||||
public func getIOCount(): Int64 {
|
|
||||||
return this.ioCount
|
|
||||||
}
|
|
||||||
|
|
||||||
public func searchByString(ipStr: String): String {
|
|
||||||
let ipBytes = parseIP(ipStr)
|
|
||||||
return this.search(ipBytes)
|
|
||||||
}
|
|
||||||
|
|
||||||
public func search(ip: Array<Byte>): String {
|
|
||||||
this.ioCount = 0
|
|
||||||
|
|
||||||
let il0 = Int64(ip[0])
|
|
||||||
let il1 = Int64(ip[1])
|
|
||||||
let idx = il0 * VectorIndexCols * VectorIndexSize + il1 * VectorIndexSize
|
|
||||||
|
|
||||||
var sPtr: Int64 = 0
|
|
||||||
var ePtr: Int64 = 0
|
|
||||||
|
|
||||||
if (this.mode == VectorIndexMode) {
|
|
||||||
sPtr = readLEUint32(this.vectorIndex, idx)
|
|
||||||
ePtr = readLEUint32(this.vectorIndex, idx + 4)
|
|
||||||
} else if (this.mode == ContentBuffMode) {
|
|
||||||
sPtr = readLEUint32(this.contentBuff, HeaderInfoLength + idx)
|
|
||||||
ePtr = readLEUint32(this.contentBuff, HeaderInfoLength + idx + 4)
|
|
||||||
} else {
|
|
||||||
let buff = Array<Byte>(VectorIndexSize, repeat: 0)
|
|
||||||
this.readFromFile(HeaderInfoLength + idx, buff)
|
|
||||||
sPtr = readLEUint32(buff, 0)
|
|
||||||
ePtr = readLEUint32(buff, 4)
|
|
||||||
}
|
|
||||||
|
|
||||||
if (sPtr == 0 || ePtr == 0) {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
let segIndexSize = this.version.segmentIndexSize
|
|
||||||
let bytes = this.version.bytes
|
|
||||||
let dBytes = bytes * 2
|
|
||||||
var dataLen: Int64 = 0
|
|
||||||
var dataPtr: Int64 = 0
|
|
||||||
|
|
||||||
var l: Int64 = 0
|
|
||||||
var h: Int64 = (ePtr - sPtr) / segIndexSize
|
|
||||||
let buff = Array<Byte>(segIndexSize, repeat: 0)
|
|
||||||
|
|
||||||
while (l <= h) {
|
|
||||||
let m = (l + h) / 2
|
|
||||||
let p = sPtr + m * segIndexSize
|
|
||||||
|
|
||||||
this.read(p, buff)
|
|
||||||
this.ioCount = this.ioCount + 1
|
|
||||||
|
|
||||||
if (this.compareIP(ip, buff, 0) < 0) {
|
|
||||||
h = m - 1
|
|
||||||
} else if (this.compareIP(ip, buff, bytes) > 0) {
|
|
||||||
l = m + 1
|
|
||||||
} else {
|
|
||||||
dataLen = readLEUint16(buff, dBytes)
|
|
||||||
dataPtr = readLEUint32(buff, dBytes + 2)
|
|
||||||
break
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
if (dataLen == 0) {
|
|
||||||
return ""
|
|
||||||
}
|
|
||||||
|
|
||||||
let regionBuff = Array<Byte>(dataLen, repeat: 0)
|
|
||||||
this.read(dataPtr, regionBuff)
|
|
||||||
return String.fromUtf8(regionBuff)
|
|
||||||
}
|
|
||||||
|
|
||||||
func read(offset: Int64, buff: Array<Byte>) {
|
|
||||||
if (this.mode == ContentBuffMode) {
|
|
||||||
for (i in 0..buff.size) {
|
|
||||||
buff[i] = this.contentBuff[offset + i]
|
|
||||||
}
|
|
||||||
} else {
|
|
||||||
this.readFromFile(offset, buff)
|
|
||||||
()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
func readFromFile(offset: Int64, buff: Array<Byte>) {
|
|
||||||
this.handle.seek(SeekPosition.Begin(offset))
|
|
||||||
this.handle.read(buff)
|
|
||||||
()
|
|
||||||
}
|
|
||||||
|
|
||||||
func compareIP(ip: Array<Byte>, entry: Array<Byte>, offset: Int64): Int64 {
|
|
||||||
if (this.version.id == IPv4VersionNo) {
|
|
||||||
return compareIPv4(ip, entry, offset)
|
|
||||||
}
|
|
||||||
return compareIPv6(ip, entry, offset)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,108 +0,0 @@
|
||||||
package ip2region.xdb
|
|
||||||
|
|
||||||
import std.net.IPAddress
|
|
||||||
|
|
||||||
// --- IP address parsing and formatting
|
|
||||||
|
|
||||||
// Parse IP string to byte array
|
|
||||||
// For IPv4, returns 4 bytes in big-endian order (network byte order)
|
|
||||||
// For IPv6, returns 16 bytes in big-endian order
|
|
||||||
public func parseIP(ip: String): Array<Byte> {
|
|
||||||
let addr = IPAddress.parse(ip)
|
|
||||||
return addr.getAddressBytes()
|
|
||||||
}
|
|
||||||
|
|
||||||
// Convert IP byte array back to string
|
|
||||||
public func ipToString(ip: Array<Byte>): String {
|
|
||||||
if (ip.size == 4) {
|
|
||||||
return "${Int64(ip[0])}.${Int64(ip[1])}.${Int64(ip[2])}.${Int64(ip[3])}"
|
|
||||||
}
|
|
||||||
// IPv6 simplified format
|
|
||||||
var result = ""
|
|
||||||
for (i in 0..ip.size) {
|
|
||||||
if (i > 0 && i % 2 == 0) {
|
|
||||||
result = result + ":"
|
|
||||||
}
|
|
||||||
result = result + "${Int64(ip[i])}"
|
|
||||||
}
|
|
||||||
return result
|
|
||||||
}
|
|
||||||
|
|
||||||
// Compare two IP byte arrays.
|
|
||||||
// Returns: -1 if ip1 < ip2, 0 if equal, 1 if ip1 > ip2
|
|
||||||
public func ipCompare(ip1: Array<Byte>, ip2: Array<Byte>): Int64 {
|
|
||||||
var len = ip1.size
|
|
||||||
if (ip2.size < len) {
|
|
||||||
len = ip2.size
|
|
||||||
}
|
|
||||||
for (i in 0..len) {
|
|
||||||
if (ip1[i] < ip2[i]) {
|
|
||||||
return -1
|
|
||||||
} else if (ip1[i] > ip2[i]) {
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (ip1.size < ip2.size) {
|
|
||||||
return -1
|
|
||||||
} else if (ip1.size > ip2.size) {
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// --- Binary data reading helpers for xdb format
|
|
||||||
|
|
||||||
// Read a little-endian UInt32 from a byte buffer at the given offset
|
|
||||||
public func readLEUint32(buf: Array<Byte>, offset: Int64): Int64 {
|
|
||||||
let b0 = Int64(buf[offset])
|
|
||||||
let b1 = Int64(buf[offset + 1]) << 8
|
|
||||||
let b2 = Int64(buf[offset + 2]) << 16
|
|
||||||
let b3 = Int64(buf[offset + 3]) << 24
|
|
||||||
return b0 | b1 | b2 | b3
|
|
||||||
}
|
|
||||||
|
|
||||||
// Read a little-endian UInt16 from a byte buffer at the given offset
|
|
||||||
public func readLEUint16(buf: Array<Byte>, offset: Int64): Int64 {
|
|
||||||
let b0 = Int64(buf[offset])
|
|
||||||
let b1 = Int64(buf[offset + 1]) << 8
|
|
||||||
return b0 | b1
|
|
||||||
}
|
|
||||||
|
|
||||||
// Compare IPv4 address (big-endian) with xdb stored IPv4 (little-endian)
|
|
||||||
// xdb stores IPv4 in little-endian, so we compare ip[i] with xdbEntry[3-i]
|
|
||||||
public func compareIPv4(ip: Array<Byte>, xdbEntry: Array<Byte>, entryOffset: Int64): Int64 {
|
|
||||||
for (i in 0..4) {
|
|
||||||
let ipByte = Int64(ip[i])
|
|
||||||
let xdbByte = Int64(xdbEntry[entryOffset + 3 - i])
|
|
||||||
if (ipByte < xdbByte) {
|
|
||||||
return -1
|
|
||||||
} else if (ipByte > xdbByte) {
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
||||||
// Load the vector index from a content buffer
|
|
||||||
// The vector index starts at HeaderInfoLength and has VectorIndexRows * VectorIndexCols * VectorIndexSize bytes
|
|
||||||
public func loadVectorIndex(cBuff: Array<Byte>): Array<Byte> {
|
|
||||||
let start = HeaderInfoLength
|
|
||||||
let len = VectorIndexRows * VectorIndexCols * VectorIndexSize
|
|
||||||
let vi = Array<Byte>(len, repeat: 0)
|
|
||||||
for (i in 0..len) {
|
|
||||||
vi[i] = cBuff[start + i]
|
|
||||||
}
|
|
||||||
return vi
|
|
||||||
}
|
|
||||||
|
|
||||||
// Compare IPv6 address (big-endian) with xdb stored IPv6 (big-endian)
|
|
||||||
public func compareIPv6(ip: Array<Byte>, xdbEntry: Array<Byte>, entryOffset: Int64): Int64 {
|
|
||||||
for (i in 0..16) {
|
|
||||||
if (ip[i] < xdbEntry[entryOffset + i]) {
|
|
||||||
return -1
|
|
||||||
} else if (ip[i] > xdbEntry[entryOffset + i]) {
|
|
||||||
return 1
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return 0
|
|
||||||
}
|
|
||||||
|
|
@ -1,41 +0,0 @@
|
||||||
package ip2region.xdb
|
|
||||||
|
|
||||||
// IP version definitions for ip2region xdb searcher
|
|
||||||
public class Version {
|
|
||||||
public let id: Int64
|
|
||||||
public let name: String
|
|
||||||
public let bytes: Int64
|
|
||||||
public let segmentIndexSize: Int64
|
|
||||||
|
|
||||||
public init(id: Int64, name: String, bytes: Int64, segmentIndexSize: Int64) {
|
|
||||||
this.id = id
|
|
||||||
this.name = name
|
|
||||||
this.bytes = bytes
|
|
||||||
this.segmentIndexSize = segmentIndexSize
|
|
||||||
}
|
|
||||||
|
|
||||||
public func toString(): String {
|
|
||||||
return "{id: ${this.id}, name: ${this.name}, bytes: ${this.bytes}, " +
|
|
||||||
"segment_index_size: ${this.segmentIndexSize}}"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
public let IPv4VersionNo: Int64 = 4
|
|
||||||
public let IPv6VersionNo: Int64 = 6
|
|
||||||
|
|
||||||
// Pre-defined IP versions
|
|
||||||
public let IPv4 = Version(4, "IPv4", 4, 14)
|
|
||||||
public let IPv6 = Version(6, "IPv6", 16, 38)
|
|
||||||
|
|
||||||
// Determine IP version from header info
|
|
||||||
public func versionFromHeader(header: Header): Version {
|
|
||||||
// Old structure (2.0) with IPv4 only
|
|
||||||
if (Int64(header.version) == Structure20) {
|
|
||||||
return IPv4
|
|
||||||
}
|
|
||||||
// Structure 3.0+
|
|
||||||
if (Int64(header.ipVersion) == IPv4VersionNo) {
|
|
||||||
return IPv4
|
|
||||||
}
|
|
||||||
return IPv6
|
|
||||||
}
|
|
||||||
|
|
@ -1 +0,0 @@
|
||||||
bin/
|
|
||||||
|
|
@ -1,27 +1,11 @@
|
||||||
|
|
||||||
all: bin header search bench make edit
|
all: xdb_search xdb_bench
|
||||||
|
|
||||||
FILES=$(wildcard src/*.cc)
|
xdb_search: xdb_search.cc xdb_search_test.cc
|
||||||
|
g++ -std=c++11 -O2 $^ -o $@
|
||||||
|
|
||||||
bin:
|
xdb_bench: xdb_search.cc xdb_bench.cc xdb_bench_test.cc
|
||||||
mkdir -p bin
|
g++ -std=c++11 -O2 $^ -o $@
|
||||||
|
|
||||||
header: $(FILES) test/header.cc
|
|
||||||
g++ -std=c++11 -O2 $^ -o bin/$@
|
|
||||||
|
|
||||||
search: $(FILES) test/search.cc
|
|
||||||
g++ -std=c++11 -O2 $^ -o bin/$@
|
|
||||||
|
|
||||||
bench: $(FILES) test/bench.cc
|
|
||||||
g++ -std=c++11 -O2 $^ -o bin/$@
|
|
||||||
|
|
||||||
make: $(FILES) test/make.cc
|
|
||||||
g++ -std=c++11 -O2 $^ -o bin/$@
|
|
||||||
|
|
||||||
edit: $(FILES)
|
|
||||||
g++ -std=c++11 -O2 $^ test/edit_v4.cc -o bin/edit_v4
|
|
||||||
g++ -std=c++11 -O2 $^ test/edit_v6.cc -o bin/edit_v6
|
|
||||||
|
|
||||||
clean:
|
clean:
|
||||||
rm -rf bin
|
rm -f xdb_search xdb_bench
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,137 +0,0 @@
|
||||||
:globe_with_meridians: [中文简体](README_zh.md) | [English](README.md)
|
|
||||||
|
|
||||||
# ip2region C++ query client
|
|
||||||
|
|
||||||
## 0. File Description
|
|
||||||
|
|
||||||
```
|
|
||||||
Makefile --------- Build
|
|
||||||
|
|
||||||
src ------------------ Source directory
|
|
||||||
src/base.* ----------- Constants and utility functions
|
|
||||||
src/ip.* ------------- IP processing implementation
|
|
||||||
src/header.* --------- xdb header parsing implementation
|
|
||||||
src/search.* --------- xdb search implementation
|
|
||||||
src/bench.* ---------- Search benchmarking implementation
|
|
||||||
src/make.* ----------- xdb file generation implementation
|
|
||||||
src/edit.* ----------- Raw data editing implementation
|
|
||||||
|
|
||||||
test ---------------- Test directory
|
|
||||||
test/header.cc ------ Test header
|
|
||||||
test/search.cc ------ Test search
|
|
||||||
test/bench.cc ------- Benchmarking
|
|
||||||
test/make.cc -------- Generate xdb file
|
|
||||||
test/edit_v4.cc ----- Test raw data editing (ipv4)
|
|
||||||
test/edit_v6.cc ----- Test raw data editing (ipv6)
|
|
||||||
|
|
||||||
|
|
||||||
bin --------------- Executable directory (generated via make)
|
|
||||||
bin/header -------- Test header
|
|
||||||
bin/search -------- Test search
|
|
||||||
bin/bench --------- Benchmarking
|
|
||||||
bin/make ---------- Generate xdb file
|
|
||||||
bin/edit_v4 ------- Test raw data editing (ipv4)
|
|
||||||
bin/edit_v6 ------- Test raw data editing (ipv6)
|
|
||||||
|
|
||||||
readme.md --------- readme
|
|
||||||
|
|
||||||
```
|
|
||||||
|
|
||||||
## 1. Compilation
|
|
||||||
|
|
||||||
```
|
|
||||||
$ make
|
|
||||||
```
|
|
||||||
|
|
||||||
## 2. Search
|
|
||||||
|
|
||||||
### 2.1 Example
|
|
||||||
|
|
||||||
```cpp
|
|
||||||
#include "src/search.h"
|
|
||||||
|
|
||||||
// IP Version: xdb::ipv4 xdb::ipv6
|
|
||||||
// Policy: xdb::policy_file xdb::policy_vector xdb::policy_content
|
|
||||||
// No cache Partial cache Full cache
|
|
||||||
int main() {
|
|
||||||
std::string xdb_name = "../../data/ip2region_v6.xdb";
|
|
||||||
int version = xdb::ipv6;
|
|
||||||
int policy = xdb::policy_content;
|
|
||||||
std::string ip = "2001:200:124::";
|
|
||||||
|
|
||||||
xdb::search_t s(xdb_name, version, policy);
|
|
||||||
std::cout << s.search(ip) << std::endl;
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
// $ g++ src/*.cc 1.cc --- Compile
|
|
||||||
// $ ./a.out ------------- Test
|
|
||||||
// Japan|Tokyo|Asagaya-minami|WIDE Project|JP
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2.2 Test xdb Header
|
|
||||||
|
|
||||||
```
|
|
||||||
$ ./bin/header
|
|
||||||
Test IPv4
|
|
||||||
Version: 3
|
|
||||||
Cache Policy: 1
|
|
||||||
File Generation Time: 2025-09-06 02:24:16
|
|
||||||
Index Start Address: 955933
|
|
||||||
Index End Address: 11042415
|
|
||||||
IP Version: 4
|
|
||||||
Pointer Bytes: 4
|
|
||||||
|
|
||||||
Test IPv6
|
|
||||||
Version: 3
|
|
||||||
Cache Policy: 1
|
|
||||||
File Generation Time: 2025-10-17 04:41:04
|
|
||||||
Index Start Address: 3094259
|
|
||||||
Index End Address: 36258303
|
|
||||||
IP Version: 6
|
|
||||||
Pointer Bytes: 4
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2.3 Test Search
|
|
||||||
|
|
||||||
```
|
|
||||||
$ ./bin/search
|
|
||||||
Test IPv4 No cache: Success
|
|
||||||
Test IPv4 Partial cache: Success
|
|
||||||
Test IPv4 Full cache: Success
|
|
||||||
Test IPv6 No cache: Success
|
|
||||||
Test IPv6 Partial cache: Success
|
|
||||||
Test IPv6 Full cache: Success
|
|
||||||
```
|
|
||||||
|
|
||||||
## 3. Benchmarking and Correctness Verification
|
|
||||||
|
|
||||||
```
|
|
||||||
./bin/bench
|
|
||||||
Test IPv4, No cache, total: 3910284, took: 27.60s, cost: 6.59μs/op, io count: 28227147
|
|
||||||
Test IPv4, Partial cache, total: 3910284, took: 21.85s, cost: 5.15μs/op, io count: 24316863
|
|
||||||
Test IPv4, Full cache, total: 3910284, took: 2.26s, cost: 0.25μs/op, io count: 0
|
|
||||||
Test IPv6, No cache, total: 4792520, took: 100.40s, cost: 20.22μs/op, io count: 80758866
|
|
||||||
Test IPv6, Partial cache, total: 4792520, took: 93.06s, cost: 18.71μs/op, io count: 75966346
|
|
||||||
Test IPv6, Full cache, total: 4792520, took: 6.24s, cost: 0.81μs/op, io count: 0
|
|
||||||
```
|
|
||||||
|
|
||||||
## 4. Generate xdb File
|
|
||||||
|
|
||||||
### 4.1 Generate xdb File
|
|
||||||
|
|
||||||
```
|
|
||||||
$ ./bin/make
|
|
||||||
Generate ipv4 xdb file, took: 0.57s
|
|
||||||
Generate ipv6 xdb file, took: 1.24s
|
|
||||||
```
|
|
||||||
|
|
||||||
## 5. Raw Data Editing
|
|
||||||
|
|
||||||
### 5.1. Instructions for Use
|
|
||||||
|
|
||||||
* New IP attribution files can contain empty lines
|
|
||||||
* New IP attribution files can be out of order; the program will automatically sort them
|
|
||||||
* New IP attribution files can overlap; as long as there is no ambiguity, the program will automatically merge them
|
|
||||||
* The final result will automatically merge adjacent lines with the same attribution
|
|
||||||
* For the following tests, the original file uses the data file provided in the repository, and the new file uses 1.txt in the current directory
|
|
||||||
|
|
@ -1,125 +0,0 @@
|
||||||
:globe_with_meridians: [中文简体](README_zh.md) | [English](README.md)
|
|
||||||
|
|
||||||
# ip2region C++ 查询客户端
|
|
||||||
|
|
||||||
## 0. 文件说明
|
|
||||||
```
|
|
||||||
Makefile --------- 构建
|
|
||||||
|
|
||||||
src ------------------ 源文件目录
|
|
||||||
src/base.* ----------- 常量及工具函数
|
|
||||||
src/ip.* ------------- 实现 IP 处理
|
|
||||||
src/header.* --------- 实现 xdb 头部解析
|
|
||||||
src/search.* --------- 实现 xdb 查找
|
|
||||||
src/bench.* ---------- 实现 查找 测速
|
|
||||||
src/make.* ----------- 实现 生成 xdb 文件
|
|
||||||
src/edit.* ----------- 实现 原始数据编辑
|
|
||||||
|
|
||||||
test ---------------- 测试目录
|
|
||||||
test/header.cc ------ 测试 头部
|
|
||||||
test/search.cc ------ 测试 查找
|
|
||||||
test/bench.cc ------- 测速
|
|
||||||
test/make.cc -------- 生成 xdb 文件
|
|
||||||
test/edit_v4.cc ----- 测试 原始数据编辑(ipv4)
|
|
||||||
test/edit_v6.cc ----- 测试 原始数据编辑(ipv6)
|
|
||||||
|
|
||||||
|
|
||||||
bin --------------- 可执行文件目录(通过 make 生成)
|
|
||||||
bin/header -------- 测试 头部
|
|
||||||
bin/search -------- 测试 查找
|
|
||||||
bin/bench --------- 测速
|
|
||||||
bin/make ---------- 生成 xdb 文件
|
|
||||||
bin/edit_v4 ------- 测试 原始数据编辑(ipv4)
|
|
||||||
bin/edit_v6 ------- 测试 原始数据编辑(ipv6)
|
|
||||||
|
|
||||||
readme.md --------- readme
|
|
||||||
```
|
|
||||||
|
|
||||||
## 1. 编译
|
|
||||||
```
|
|
||||||
$ make
|
|
||||||
```
|
|
||||||
|
|
||||||
## 2. 查找
|
|
||||||
### 2.1 示例
|
|
||||||
```cpp
|
|
||||||
#include "src/search.h"
|
|
||||||
|
|
||||||
// IP 版本: xdb::ipv4 xdb::ipv6
|
|
||||||
// 策略: xdb::policy_file xdb::policy_vector xdb::policy_content
|
|
||||||
// 不缓存 部分缓存 全部缓存
|
|
||||||
int main() {
|
|
||||||
std::string xdb_name = "../../data/ip2region_v6.xdb";
|
|
||||||
int version = xdb::ipv6;
|
|
||||||
int policy = xdb::policy_content;
|
|
||||||
std::string ip = "2001:200:124::";
|
|
||||||
|
|
||||||
xdb::search_t s(xdb_name, version, policy);
|
|
||||||
std::cout << s.search(ip) << std::endl;
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
// $ g++ src/*.cc 1.cc --- 编译
|
|
||||||
// $ ./a.out ------------- 测试
|
|
||||||
// Japan|Tokyo|Asagaya-minami|WIDE Project|JP
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2.2 测试 xdb 头部
|
|
||||||
```
|
|
||||||
$ ./bin/header
|
|
||||||
测试 IPv4
|
|
||||||
版本号: 3
|
|
||||||
缓存策略: 1
|
|
||||||
文件生成时间: 2025-09-06 02:24:16
|
|
||||||
索引起始地址: 955933
|
|
||||||
索引结束地址: 11042415
|
|
||||||
IP版本: 4
|
|
||||||
指针字节数: 4
|
|
||||||
|
|
||||||
测试 IPv6
|
|
||||||
版本号: 3
|
|
||||||
缓存策略: 1
|
|
||||||
文件生成时间: 2025-10-17 04:41:04
|
|
||||||
索引起始地址: 3094259
|
|
||||||
索引结束地址: 36258303
|
|
||||||
IP版本: 6
|
|
||||||
指针字节数: 4
|
|
||||||
```
|
|
||||||
|
|
||||||
### 2.3 测试查找
|
|
||||||
```
|
|
||||||
$ ./bin/search
|
|
||||||
测试 IPv4 不缓存: 成功
|
|
||||||
测试 IPv4 部分缓存: 成功
|
|
||||||
测试 IPv4 全部缓存: 成功
|
|
||||||
测试 IPv6 不缓存: 成功
|
|
||||||
测试 IPv6 部分缓存: 成功
|
|
||||||
测试 IPv6 全部缓存: 成功
|
|
||||||
```
|
|
||||||
|
|
||||||
## 3. 测速以及检验正确性
|
|
||||||
```
|
|
||||||
./bin/bench
|
|
||||||
测试 IPv4, 不缓存, total: 3910284, took: 27.60s, cost: 6.59μs/op, io count: 28227147
|
|
||||||
测试 IPv4, 部分缓存, total: 3910284, took: 21.85s, cost: 5.15μs/op, io count: 24316863
|
|
||||||
测试 IPv4, 全部缓存, total: 3910284, took: 2.26s, cost: 0.25μs/op, io count: 0
|
|
||||||
测试 IPv6, 不缓存, total: 4792520, took: 100.40s, cost: 20.22μs/op, io count: 80758866
|
|
||||||
测试 IPv6, 部分缓存, total: 4792520, took: 93.06s, cost: 18.71μs/op, io count: 75966346
|
|
||||||
测试 IPv6, 全部缓存, total: 4792520, took: 6.24s, cost: 0.81μs/op, io count: 0
|
|
||||||
```
|
|
||||||
|
|
||||||
## 4. 生成 xdb 文件
|
|
||||||
### 4.1 生成 xdb 文件
|
|
||||||
```
|
|
||||||
$ ./bin/make
|
|
||||||
生成 ipv4 的 xdb 文件, took: 0.57s
|
|
||||||
生成 ipv6 的 xdb 文件, took: 1.24s
|
|
||||||
```
|
|
||||||
|
|
||||||
## 5. 原始数据编辑
|
|
||||||
### 5.1. 使用说明
|
|
||||||
* 新的IP归属地文件可以包含空行
|
|
||||||
* 新的IP归属地文件顺序可以乱序, 程序会自动排序
|
|
||||||
* 新的IP归属地文件顺序可以重叠, 只要无二义性, 程序会自动合并
|
|
||||||
* 最终的结果会将相邻的且归属地相同的行自动合并
|
|
||||||
* 以下测试, 原文件使用仓库自带的数据文件, 新文件使用当前目录下的 1.txt
|
|
||||||
|
|
@ -0,0 +1,108 @@
|
||||||
|
# ip2region xdb C++ 查询客户端实现
|
||||||
|
|
||||||
|
## 使用方式
|
||||||
|
### 完全基于文件的查询
|
||||||
|
```
|
||||||
|
#include <iostream>
|
||||||
|
|
||||||
|
#include "xdb_search.h"
|
||||||
|
|
||||||
|
int main(int argc, char* argv[]) {
|
||||||
|
char file_name[] = "../../data/ip2region.xdb";
|
||||||
|
char ip[] = "1.2.3.4";
|
||||||
|
|
||||||
|
xdb_search_t xdb(file_name);
|
||||||
|
xdb.init_file();
|
||||||
|
|
||||||
|
std::cout << xdb.search(ip) << std::endl;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 缓存 `vector_index` 索引
|
||||||
|
```
|
||||||
|
#include <iostream>
|
||||||
|
|
||||||
|
#include "xdb_search.h"
|
||||||
|
|
||||||
|
int main(int argc, char* argv[]) {
|
||||||
|
char file_name[] = "../../data/ip2region.xdb";
|
||||||
|
char ip[] = "1.2.3.4";
|
||||||
|
|
||||||
|
xdb_search_t xdb(file_name);
|
||||||
|
xdb.init_vector_index();
|
||||||
|
|
||||||
|
std::cout << xdb.search(ip) << std::endl;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### 缓存整个 `xdb` 数据
|
||||||
|
```
|
||||||
|
#include <iostream>
|
||||||
|
|
||||||
|
#include "xdb_search.h"
|
||||||
|
|
||||||
|
int main(int argc, char* argv[]) {
|
||||||
|
char file_name[] = "../../data/ip2region.xdb";
|
||||||
|
char ip[] = "1.2.3.4";
|
||||||
|
|
||||||
|
xdb_search_t xdb(file_name);
|
||||||
|
xdb.init_content();
|
||||||
|
|
||||||
|
std::cout << xdb.search(ip) << std::endl;
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## 测试程序编译
|
||||||
|
1. 切换到当前目录
|
||||||
|
2. 编译
|
||||||
|
|
||||||
|
```
|
||||||
|
$ make
|
||||||
|
g++ -std=c++11 -O2 xdb_search.cc xdb_search_test.cc -o xdb_search
|
||||||
|
g++ -std=c++11 -O2 xdb_search.cc xdb_bench.cc xdb_bench_test.cc -o xdb_bench
|
||||||
|
```
|
||||||
|
|
||||||
|
## 测试查询
|
||||||
|
### 说明
|
||||||
|
```
|
||||||
|
$ ./xdb_search --help
|
||||||
|
./xdb_search [command options]
|
||||||
|
options:
|
||||||
|
--db string ip2region binary xdb file path
|
||||||
|
--cache-policy string cache policy: file/vector_index/content
|
||||||
|
--help print help
|
||||||
|
```
|
||||||
|
|
||||||
|
### 测试
|
||||||
|
```
|
||||||
|
$ ./xdb_search --db ../../data/ip2region.xdb --cache-policy vector_index
|
||||||
|
cache policy : vector_index
|
||||||
|
ip2region>> 1.2.3.4
|
||||||
|
美国|0|华盛顿|0|谷歌
|
||||||
|
```
|
||||||
|
|
||||||
|
## bench 测试
|
||||||
|
### 说明
|
||||||
|
```
|
||||||
|
$ ./xdb_bench --help
|
||||||
|
./xdb_bench [command options]
|
||||||
|
options:
|
||||||
|
--db string ip2region binary xdb file path
|
||||||
|
--src string source ip text file path
|
||||||
|
--cache-policy string cache policy: file/vector_index/content
|
||||||
|
--help print help
|
||||||
|
```
|
||||||
|
|
||||||
|
### 测试
|
||||||
|
```
|
||||||
|
$ ./xdb_bench --db ../../data/ip2region.xdb --src ../../data/ip.merge.txt --cache-policy content
|
||||||
|
total: 3419220, took: 3.44 s, cost: 0.27 μs/op, io count: 0
|
||||||
|
$ ./xdb_bench --db ../../data/ip2region.xdb --src ../../data/ip.merge.txt --cache-policy vector_index
|
||||||
|
total: 3419220, took: 45.99 s, cost: 12.24 μs/op, io count: 21739300
|
||||||
|
$ ./xdb_bench --db ../../data/ip2region.xdb --src ../../data/ip.merge.txt --cache-policy file
|
||||||
|
total: 3419220, took: 60.39 s, cost: 16.32 μs/op, io count: 25158520
|
||||||
|
```
|
||||||
|
|
||||||
|
|
@ -1,74 +0,0 @@
|
||||||
|
|
||||||
#include "base.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
int ip_version; // ip 版本
|
|
||||||
int ip_size; // ip 占的字节数
|
|
||||||
int content_size;
|
|
||||||
|
|
||||||
void init_xdb(int version) {
|
|
||||||
ip_version = version;
|
|
||||||
ip_size = version == ipv4 ? 4 : 16;
|
|
||||||
content_size = ip_size * 2 + 2 + 4;
|
|
||||||
}
|
|
||||||
|
|
||||||
void log_exit(const string &msg) {
|
|
||||||
std::cout << msg << std::endl;
|
|
||||||
exit(-1);
|
|
||||||
}
|
|
||||||
|
|
||||||
void read_bin(int index, char *buf, size_t len, FILE *db) {
|
|
||||||
fseek(db, index, SEEK_SET);
|
|
||||||
if (fread(buf, 1, len, db) != len)
|
|
||||||
log_exit(__func__);
|
|
||||||
}
|
|
||||||
|
|
||||||
unsigned to_uint(const char *buf) {
|
|
||||||
return ((buf[0]) & 0x000000FF) | ((buf[1] << 8) & 0x0000FF00) |
|
|
||||||
((buf[2] << 16) & 0x00FF0000) | ((buf[3] << 24) & 0xFF000000);
|
|
||||||
}
|
|
||||||
|
|
||||||
unsigned to_ushort(const char *buf) {
|
|
||||||
return ((buf[0]) & 0x000000FF) | ((buf[1] << 8) & 0x0000FF00);
|
|
||||||
}
|
|
||||||
|
|
||||||
unsigned to_int(const char *buf, int n) {
|
|
||||||
return n == 2 ? to_ushort(buf) : to_uint(buf);
|
|
||||||
}
|
|
||||||
|
|
||||||
void write_uint(unsigned data, char buf[]) {
|
|
||||||
buf[0] = (data >> 0) & 0xFF;
|
|
||||||
buf[1] = (data >> 8) & 0xFF;
|
|
||||||
buf[2] = (data >> 16) & 0xFF;
|
|
||||||
buf[3] = (data >> 24) & 0xFF;
|
|
||||||
}
|
|
||||||
|
|
||||||
void write_uint(unsigned data, FILE *dst) {
|
|
||||||
char buf[4];
|
|
||||||
write_uint(data, buf);
|
|
||||||
fwrite(buf, 1, sizeof(buf), dst);
|
|
||||||
}
|
|
||||||
|
|
||||||
void write_ushort(unsigned data, char buf[]) {
|
|
||||||
buf[0] = (data >> 0) & 0xFF;
|
|
||||||
buf[1] = (data >> 8) & 0xFF;
|
|
||||||
}
|
|
||||||
|
|
||||||
void write_ushort(unsigned data, FILE *dst) {
|
|
||||||
char buf[2];
|
|
||||||
write_ushort(data, buf);
|
|
||||||
fwrite(buf, 1, sizeof(buf), dst);
|
|
||||||
}
|
|
||||||
|
|
||||||
void write_string(const char *buf, unsigned len, FILE *dst) {
|
|
||||||
fwrite(buf, 1, len, dst);
|
|
||||||
}
|
|
||||||
|
|
||||||
unsigned long long get_time() {
|
|
||||||
struct timeval tv1;
|
|
||||||
gettimeofday(&tv1, NULL);
|
|
||||||
return (unsigned long long)tv1.tv_sec * 1000 * 1000 + tv1.tv_usec;
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
@ -1,58 +0,0 @@
|
||||||
#ifndef BASE_H
|
|
||||||
#define BASE_H
|
|
||||||
|
|
||||||
#include <arpa/inet.h>
|
|
||||||
#include <stdio.h>
|
|
||||||
#include <stdlib.h>
|
|
||||||
#include <string.h>
|
|
||||||
#include <sys/time.h>
|
|
||||||
|
|
||||||
#include <algorithm>
|
|
||||||
#include <iostream>
|
|
||||||
#include <list>
|
|
||||||
#include <map>
|
|
||||||
#include <string>
|
|
||||||
#include <unordered_map>
|
|
||||||
#include <vector>
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
using std::string;
|
|
||||||
|
|
||||||
constexpr int ipv4 = 4;
|
|
||||||
constexpr int ipv6 = 6;
|
|
||||||
|
|
||||||
constexpr int policy_file = 0;
|
|
||||||
constexpr int policy_vector = 1;
|
|
||||||
constexpr int policy_content = 2;
|
|
||||||
|
|
||||||
constexpr int length_header = 256;
|
|
||||||
constexpr int length_vector = 256 * 256 * 8;
|
|
||||||
|
|
||||||
extern int ip_version; // ip 版本
|
|
||||||
extern int ip_size; // ip 占的字节数
|
|
||||||
extern int content_size;
|
|
||||||
|
|
||||||
void init_xdb(int version);
|
|
||||||
|
|
||||||
void log_exit(const string &msg);
|
|
||||||
|
|
||||||
void read_bin(int index, char *buf, size_t len, FILE *db);
|
|
||||||
|
|
||||||
unsigned to_uint(const char *buf);
|
|
||||||
unsigned to_ushort(const char *buf);
|
|
||||||
unsigned to_int(const char *buf, int n);
|
|
||||||
|
|
||||||
void write_uint(unsigned data, char buf[]);
|
|
||||||
void write_uint(unsigned data, FILE *dst);
|
|
||||||
|
|
||||||
void write_ushort(unsigned data, char buf[]);
|
|
||||||
void write_ushort(unsigned data, FILE *dst);
|
|
||||||
|
|
||||||
void write_string(const char *buf, unsigned len, FILE *dst);
|
|
||||||
|
|
||||||
unsigned long long get_time();
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
||||||
#endif
|
|
||||||
|
|
@ -1,65 +0,0 @@
|
||||||
|
|
||||||
#include "bench.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
bench_t::bench_t(const std::string &file_name, int version, int policy)
|
|
||||||
: search(file_name, version, policy) {
|
|
||||||
}
|
|
||||||
|
|
||||||
void bench_t::test_one(const ip_t &ip, const string region) {
|
|
||||||
if (search.search(ip.to_string()) != region)
|
|
||||||
xdb::log_exit("failed: " + ip.to_string() + " " + region);
|
|
||||||
sum_io_count += search.get_io_count();
|
|
||||||
sum_cost_time += search.get_cost_time();
|
|
||||||
sum_count++;
|
|
||||||
}
|
|
||||||
|
|
||||||
void bench_t::test_line(char *buf) {
|
|
||||||
size_t buf_len = strlen(buf);
|
|
||||||
if (buf_len == 0)
|
|
||||||
return;
|
|
||||||
buf[buf_len - 1] = '\0'; // 去掉换行符
|
|
||||||
|
|
||||||
node_t node(buf);
|
|
||||||
|
|
||||||
// 只测五个
|
|
||||||
for (int i = 0; i < 5 && node.ip1 < node.ip2; ++i) {
|
|
||||||
test_one(node.ip1, node.region);
|
|
||||||
node.ip1 = node.ip1 + 1;
|
|
||||||
}
|
|
||||||
test_one(node.ip2, node.region);
|
|
||||||
}
|
|
||||||
|
|
||||||
void bench_t::test_file(const std::string &file_name) {
|
|
||||||
FILE *f = fopen(file_name.data(), "r");
|
|
||||||
if (f == NULL)
|
|
||||||
xdb::log_exit("can't open " + file_name);
|
|
||||||
char buf[1024];
|
|
||||||
while (fgets(buf, sizeof(buf), f) != NULL)
|
|
||||||
test_line(buf);
|
|
||||||
}
|
|
||||||
|
|
||||||
void bench_t::test(const string &file_name) {
|
|
||||||
sum_io_count = 0;
|
|
||||||
sum_cost_time = 0;
|
|
||||||
sum_count = 0;
|
|
||||||
|
|
||||||
unsigned long long tv1 = xdb::get_time();
|
|
||||||
test_file(file_name);
|
|
||||||
unsigned long long tv2 = xdb::get_time();
|
|
||||||
|
|
||||||
double took = (tv2 - tv1) * 1.0 / 1000 / 1000;
|
|
||||||
double cost = sum_cost_time * 1.0 / sum_count;
|
|
||||||
|
|
||||||
printf(
|
|
||||||
"total: %llu, took: %8.2fs, cost: %6.2fμs/op, io "
|
|
||||||
"count: "
|
|
||||||
"%llu\n",
|
|
||||||
sum_count,
|
|
||||||
took,
|
|
||||||
cost,
|
|
||||||
sum_io_count);
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
#ifndef BENCH_H
|
|
||||||
#define BENCH_H
|
|
||||||
|
|
||||||
#include "search.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
class bench_t {
|
|
||||||
public:
|
|
||||||
bench_t(const string &file_name, int version, int policy);
|
|
||||||
|
|
||||||
void test(const string &file_name);
|
|
||||||
|
|
||||||
private:
|
|
||||||
void test_one(const ip_t &ip, const string region);
|
|
||||||
void test_line(char *buf);
|
|
||||||
void test_file(const std::string &file_name);
|
|
||||||
|
|
||||||
search_t search;
|
|
||||||
|
|
||||||
unsigned long long sum_io_count;
|
|
||||||
unsigned long long sum_cost_time;
|
|
||||||
unsigned long long sum_count;
|
|
||||||
};
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
#endif
|
|
||||||
|
|
@ -1,170 +0,0 @@
|
||||||
|
|
||||||
#include "edit.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
void handle_ip_txt(const string& name, std::list<node_t>& regions) {
|
|
||||||
FILE* f = fopen(name.data(), "r");
|
|
||||||
if (f == NULL)
|
|
||||||
log_exit("can't open " + name);
|
|
||||||
|
|
||||||
char buf[1024];
|
|
||||||
while (fgets(buf, sizeof(buf), f) != NULL) {
|
|
||||||
unsigned int buf_len = strlen(buf);
|
|
||||||
// 去掉多余的空
|
|
||||||
while (buf_len > 0 && isspace(buf[buf_len - 1]))
|
|
||||||
--buf_len;
|
|
||||||
if (buf_len == 0)
|
|
||||||
continue;
|
|
||||||
buf[buf_len] = '\0';
|
|
||||||
regions.push_back(node_t(buf));
|
|
||||||
}
|
|
||||||
|
|
||||||
fclose(f);
|
|
||||||
}
|
|
||||||
|
|
||||||
void edit_t::handle_new_file(const std::string& file_name) {
|
|
||||||
handle_ip_txt(file_name, new_regions); // 输入
|
|
||||||
new_regions.sort(); // 排序
|
|
||||||
|
|
||||||
// 检验及其去重
|
|
||||||
auto it = new_regions.begin();
|
|
||||||
|
|
||||||
for (;;) {
|
|
||||||
if (it == new_regions.end())
|
|
||||||
break;
|
|
||||||
auto next = it;
|
|
||||||
++next;
|
|
||||||
if (next == new_regions.end())
|
|
||||||
break;
|
|
||||||
if (it->ip1 > it->ip2)
|
|
||||||
it = new_regions.erase(it); // 非法, 直接跳过
|
|
||||||
else if (it->ip1 == next->ip1 || next->ip1 <= it->ip2) {
|
|
||||||
// 数据重叠
|
|
||||||
if (it->region != next->region)
|
|
||||||
log_exit("数据有二义性: " + it->to_string() + ", " +
|
|
||||||
next->to_string());
|
|
||||||
it->ip2 = std::max(it->ip2, next->ip2);
|
|
||||||
new_regions.erase(next);
|
|
||||||
} else if (it->ip2 + 1 == next->ip1 && it->region == next->region) {
|
|
||||||
// 数据连接
|
|
||||||
it->ip2 = next->ip2;
|
|
||||||
new_regions.erase(next);
|
|
||||||
} else {
|
|
||||||
++it;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
void edit_t::handle_old_file(const std::string& file_name) {
|
|
||||||
handle_ip_txt(file_name, old_regions);
|
|
||||||
}
|
|
||||||
|
|
||||||
void edit_t::merge() {
|
|
||||||
auto it1 = old_regions.begin();
|
|
||||||
auto it2 = new_regions.begin();
|
|
||||||
|
|
||||||
for (;;) {
|
|
||||||
if (it2 == new_regions.end())
|
|
||||||
break;
|
|
||||||
if (it2->ip1 > it2->ip2) {
|
|
||||||
++it2;
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
// it1->ip1 it1->ip2 it2->ip1 it2->ip2
|
|
||||||
while (it1->ip2 < it2->ip1)
|
|
||||||
++it1;
|
|
||||||
if (it1->ip2 <= it2->ip2) {
|
|
||||||
// it1->ip1 it2->ip1 it1->ip2 it2->ip2
|
|
||||||
node_t node;
|
|
||||||
node.ip1 = it2->ip1;
|
|
||||||
node.ip2 = it1->ip2;
|
|
||||||
node.region = it2->region;
|
|
||||||
|
|
||||||
it1->ip2 = node.ip1 - 1;
|
|
||||||
it2->ip1 = node.ip2 + 1;
|
|
||||||
|
|
||||||
++it1;
|
|
||||||
it1 = old_regions.insert(it1, node);
|
|
||||||
++it1;
|
|
||||||
} else {
|
|
||||||
// it1->ip1 it2->ip1 it2->ip2 it1->ip2
|
|
||||||
node_t node;
|
|
||||||
node.ip1 = it2->ip2 + 1;
|
|
||||||
node.ip2 = it1->ip2;
|
|
||||||
node.region = it1->region;
|
|
||||||
|
|
||||||
it1->ip2 = it2->ip1 - 1;
|
|
||||||
|
|
||||||
++it1;
|
|
||||||
it1 = old_regions.insert(it1, *it2);
|
|
||||||
|
|
||||||
++it1;
|
|
||||||
it1 = old_regions.insert(it1, node);
|
|
||||||
|
|
||||||
++it2;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
void edit_t::write_old_file(const std::string& file_name) {
|
|
||||||
FILE* f = fopen(file_name.data(), "w");
|
|
||||||
if (f == NULL)
|
|
||||||
log_exit("can't open " + file_name);
|
|
||||||
|
|
||||||
auto it = old_regions.begin();
|
|
||||||
|
|
||||||
// 删除非法的数据
|
|
||||||
for (;;) {
|
|
||||||
if (it == old_regions.end())
|
|
||||||
break;
|
|
||||||
if (it->ip1 > it->ip2)
|
|
||||||
it = old_regions.erase(it);
|
|
||||||
else
|
|
||||||
++it;
|
|
||||||
}
|
|
||||||
|
|
||||||
// 合并数据域相同的相邻数据
|
|
||||||
it = old_regions.begin();
|
|
||||||
for (;;) {
|
|
||||||
if (it == old_regions.end())
|
|
||||||
break;
|
|
||||||
auto next = it;
|
|
||||||
++next;
|
|
||||||
if (next == old_regions.end())
|
|
||||||
break;
|
|
||||||
if (it->region == next->region) {
|
|
||||||
it->ip2 = next->ip2;
|
|
||||||
old_regions.erase(next);
|
|
||||||
} else {
|
|
||||||
++it;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
for (auto& d : old_regions) {
|
|
||||||
string res =
|
|
||||||
d.ip1.to_string() + "|" + d.ip2.to_string() + "|" + d.region + "\n";
|
|
||||||
fputs(res.data(), f);
|
|
||||||
}
|
|
||||||
|
|
||||||
fclose(f);
|
|
||||||
}
|
|
||||||
|
|
||||||
edit_t::edit_t(const string& name_old, const string& name_new, int version) {
|
|
||||||
unsigned long long tv1 = get_time();
|
|
||||||
|
|
||||||
init_xdb(version);
|
|
||||||
|
|
||||||
handle_new_file(name_new);
|
|
||||||
handle_old_file(name_old);
|
|
||||||
merge();
|
|
||||||
write_old_file(name_old);
|
|
||||||
|
|
||||||
unsigned long long tv2 = get_time();
|
|
||||||
|
|
||||||
double took = (tv2 - tv1) * 1.0 / 1000 / 1000;
|
|
||||||
|
|
||||||
printf("took: %.2fs\n", took);
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
@ -1,24 +0,0 @@
|
||||||
#ifndef EDIT_H
|
|
||||||
#define EDIT_H
|
|
||||||
|
|
||||||
#include "ip.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
class edit_t {
|
|
||||||
public:
|
|
||||||
edit_t(const string& old_name, const string& new_name, int version);
|
|
||||||
|
|
||||||
private:
|
|
||||||
void handle_new_file(const string& file_name);
|
|
||||||
void handle_old_file(const string& file_name);
|
|
||||||
void merge();
|
|
||||||
void write_old_file(const string& file_name);
|
|
||||||
|
|
||||||
std::list<node_t> old_regions;
|
|
||||||
std::list<node_t> new_regions;
|
|
||||||
};
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
||||||
#endif
|
|
||||||
|
|
@ -1,41 +0,0 @@
|
||||||
|
|
||||||
#include "header.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
header_t::header_t(FILE* db) {
|
|
||||||
read_bin(0, header, sizeof(header), db);
|
|
||||||
}
|
|
||||||
|
|
||||||
header_t::~header_t() {
|
|
||||||
}
|
|
||||||
|
|
||||||
int header_t::version() {
|
|
||||||
return to_int(header, 2); // 版本号(2)
|
|
||||||
}
|
|
||||||
|
|
||||||
int header_t::index_policy() {
|
|
||||||
return to_int(header + 2, 2); // 缓存策略(2)
|
|
||||||
}
|
|
||||||
|
|
||||||
int header_t::create_at() {
|
|
||||||
return to_int(header + 4, 4); // 文件生成时间(4)
|
|
||||||
}
|
|
||||||
|
|
||||||
int header_t::index_start() {
|
|
||||||
return to_int(header + 8, 4); // 索引起始地址(4)
|
|
||||||
}
|
|
||||||
|
|
||||||
int header_t::index_end() {
|
|
||||||
return to_int(header + 12, 4); // 索引结束地址(4)
|
|
||||||
}
|
|
||||||
|
|
||||||
int header_t::ip_version() {
|
|
||||||
return to_int(header + 16, 2); // IP 版本(2)
|
|
||||||
}
|
|
||||||
|
|
||||||
int header_t::ptr() {
|
|
||||||
return to_int(header + 18, 2); // 指针字节数(2)
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
@ -1,27 +0,0 @@
|
||||||
#ifndef HEADER_H
|
|
||||||
#define HEADER_H
|
|
||||||
|
|
||||||
#include "base.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
class header_t {
|
|
||||||
public:
|
|
||||||
header_t(FILE* db);
|
|
||||||
virtual ~header_t();
|
|
||||||
|
|
||||||
int version(); // 版本号
|
|
||||||
int index_policy(); // 缓存策略
|
|
||||||
int create_at(); // 文件生成时间
|
|
||||||
int index_start(); // 索引起始地址
|
|
||||||
int index_end(); // 索引结束地址
|
|
||||||
int ip_version(); // IP 版本
|
|
||||||
int ptr(); // 指针字节数
|
|
||||||
|
|
||||||
protected:
|
|
||||||
char header[length_header];
|
|
||||||
};
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
||||||
#endif
|
|
||||||
|
|
@ -1,158 +0,0 @@
|
||||||
|
|
||||||
#include "ip.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
ip_t::ip_t() {
|
|
||||||
memset(p, '\0', sizeof(p));
|
|
||||||
}
|
|
||||||
|
|
||||||
ip_t::ip_t(const ip_t& rhs, int val) {
|
|
||||||
memcpy(p, rhs.p, ip_size);
|
|
||||||
|
|
||||||
if (val == 0 || val == 255)
|
|
||||||
for (int i = 2; i < ip_size; ++i)
|
|
||||||
p[i] = val;
|
|
||||||
}
|
|
||||||
|
|
||||||
ip_t::ip_t(const char* p) {
|
|
||||||
from_xdb(p);
|
|
||||||
}
|
|
||||||
|
|
||||||
bool ip_t::from_str(const string& str) {
|
|
||||||
int af_inet = ip_version == ipv4 ? AF_INET : AF_INET6;
|
|
||||||
return inet_pton(af_inet, str.data(), p) == 1;
|
|
||||||
}
|
|
||||||
|
|
||||||
void ip_t::from_xdb(const char str[16]) {
|
|
||||||
for (int i = 0; i < ip_size; ++i)
|
|
||||||
if (ip_version == ipv6)
|
|
||||||
p[i] = str[i];
|
|
||||||
else
|
|
||||||
p[i] = str[ip_size - 1 - i];
|
|
||||||
}
|
|
||||||
|
|
||||||
ip_t& ip_t::operator=(const ip_t& rhs) {
|
|
||||||
memcpy(p, rhs.p, ip_size);
|
|
||||||
return *this;
|
|
||||||
}
|
|
||||||
|
|
||||||
int ip_t::compare(const ip_t& rhs) const {
|
|
||||||
for (int i = 0; i < ip_size; ++i) {
|
|
||||||
if ((unsigned char)p[i] > (unsigned char)rhs.p[i])
|
|
||||||
return 1;
|
|
||||||
if ((unsigned char)p[i] < (unsigned char)rhs.p[i])
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool ip_t::operator<(const ip_t& rhs) const {
|
|
||||||
return compare(rhs) < 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool ip_t::operator<=(const ip_t& rhs) const {
|
|
||||||
return compare(rhs) <= 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool ip_t::operator>(const ip_t& rhs) const {
|
|
||||||
return compare(rhs) > 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool ip_t::operator>=(const ip_t& rhs) const {
|
|
||||||
return compare(rhs) >= 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool ip_t::operator==(const ip_t& rhs) const {
|
|
||||||
return compare(rhs) == 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
bool ip_t::operator!=(const ip_t& rhs) const {
|
|
||||||
return compare(rhs) != 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
string ip_t::to_string() const {
|
|
||||||
char buf[INET6_ADDRSTRLEN + 1];
|
|
||||||
int af_inet = ip_version == ipv4 ? AF_INET : AF_INET6;
|
|
||||||
inet_ntop(af_inet, p, buf, sizeof(buf));
|
|
||||||
return string(buf);
|
|
||||||
}
|
|
||||||
|
|
||||||
string ip_t::to_bit() const {
|
|
||||||
string str;
|
|
||||||
for (int i = 0; i < ip_size; ++i)
|
|
||||||
if (ip_version == ipv6)
|
|
||||||
str.push_back(p[i]);
|
|
||||||
else
|
|
||||||
str.push_back(p[ip_size - 1 - i]);
|
|
||||||
return str;
|
|
||||||
}
|
|
||||||
|
|
||||||
ip_t operator+(const ip_t& lhs, int v) {
|
|
||||||
ip_t ip;
|
|
||||||
|
|
||||||
int i = ip_size;
|
|
||||||
while (--i >= 0) {
|
|
||||||
v += lhs.p[i];
|
|
||||||
ip.p[i] = v % 256;
|
|
||||||
v /= 256;
|
|
||||||
}
|
|
||||||
return ip;
|
|
||||||
}
|
|
||||||
|
|
||||||
ip_t operator-(const ip_t& lhs, int v) {
|
|
||||||
ip_t ip;
|
|
||||||
|
|
||||||
int i = ip_size;
|
|
||||||
v = -v;
|
|
||||||
while (--i >= 0) {
|
|
||||||
v += lhs.p[i];
|
|
||||||
if (v == -1)
|
|
||||||
ip.p[i] = 255;
|
|
||||||
else {
|
|
||||||
ip.p[i] = v;
|
|
||||||
v = 0;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return ip;
|
|
||||||
}
|
|
||||||
|
|
||||||
// node_t
|
|
||||||
node_t::node_t() {
|
|
||||||
}
|
|
||||||
|
|
||||||
node_t::node_t(char* buf) {
|
|
||||||
char* pos1 = strchr(buf, '|');
|
|
||||||
|
|
||||||
if (pos1 == NULL)
|
|
||||||
log_exit("invalid data: " + std::string(buf));
|
|
||||||
char* pos2 = strchr(pos1 + 1, '|');
|
|
||||||
if (pos2 == NULL)
|
|
||||||
log_exit("invalid data: " + std::string(buf));
|
|
||||||
*pos1 = '\0';
|
|
||||||
*pos2 = '\0';
|
|
||||||
|
|
||||||
region = pos2 + 1;
|
|
||||||
|
|
||||||
if (!ip1.from_str(buf) || !ip2.from_str(pos1 + 1) || ip2 < ip1 ||
|
|
||||||
region.empty()) {
|
|
||||||
*pos1 = *pos2 = '|';
|
|
||||||
log_exit(string("invalid data: ") + buf);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
bool node_t::operator<(const node_t& rhs) const {
|
|
||||||
if (ip1 < rhs.ip1)
|
|
||||||
return true;
|
|
||||||
return ip2 < rhs.ip2;
|
|
||||||
}
|
|
||||||
|
|
||||||
string node_t::to_string() const {
|
|
||||||
return ip1.to_string() + "|" + ip2.to_string() + "|" + region;
|
|
||||||
}
|
|
||||||
|
|
||||||
string node_t::to_bit() const {
|
|
||||||
return ip1.to_bit() + ip2.to_bit();
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
@ -1,52 +0,0 @@
|
||||||
#ifndef IP_H
|
|
||||||
#define IP_H
|
|
||||||
|
|
||||||
#include "base.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
struct ip_t {
|
|
||||||
unsigned char p[16];
|
|
||||||
|
|
||||||
ip_t();
|
|
||||||
ip_t(const char* p);
|
|
||||||
// val 为 0 或 255 时, 将 ip 的后几位置为 val
|
|
||||||
ip_t(const ip_t& rhs, int val = -1);
|
|
||||||
|
|
||||||
bool from_str(const string& str);
|
|
||||||
void from_xdb(const char str[16]);
|
|
||||||
|
|
||||||
ip_t& operator=(const ip_t& rhs);
|
|
||||||
|
|
||||||
int compare(const ip_t& rhs) const;
|
|
||||||
bool operator<(const ip_t& rhs) const;
|
|
||||||
bool operator<=(const ip_t& rhs) const;
|
|
||||||
bool operator>(const ip_t& rhs) const;
|
|
||||||
bool operator>=(const ip_t& rhs) const;
|
|
||||||
bool operator==(const ip_t& rhs) const;
|
|
||||||
bool operator!=(const ip_t& rhs) const;
|
|
||||||
|
|
||||||
string to_string() const;
|
|
||||||
string to_bit() const;
|
|
||||||
};
|
|
||||||
|
|
||||||
ip_t operator+(const ip_t& lhs, int v);
|
|
||||||
ip_t operator-(const ip_t& lhs, int v);
|
|
||||||
|
|
||||||
struct node_t {
|
|
||||||
ip_t ip1;
|
|
||||||
ip_t ip2;
|
|
||||||
string region;
|
|
||||||
|
|
||||||
node_t();
|
|
||||||
node_t(char* buf);
|
|
||||||
|
|
||||||
bool operator<(const node_t& rhs) const;
|
|
||||||
|
|
||||||
string to_string() const;
|
|
||||||
string to_bit() const;
|
|
||||||
};
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
||||||
#endif
|
|
||||||
|
|
@ -1,167 +0,0 @@
|
||||||
|
|
||||||
#include "make.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
void make_t::vector_index_push_back(int row, int col, const node_t &node) {
|
|
||||||
vector_index[row][col].push_back(
|
|
||||||
std::make_pair<string, string>(node.to_bit(), string(node.region)));
|
|
||||||
}
|
|
||||||
|
|
||||||
void make_t::vector_index_push_back(node_t &node) {
|
|
||||||
ip_t ip1 = node.ip1;
|
|
||||||
ip_t ip2 = node.ip2;
|
|
||||||
|
|
||||||
unsigned ip1_1 = ip1.p[0];
|
|
||||||
unsigned ip1_2 = ip1.p[1];
|
|
||||||
unsigned ip2_1 = ip2.p[0];
|
|
||||||
unsigned ip2_2 = ip2.p[1];
|
|
||||||
|
|
||||||
if (ip1_1 == ip2_1 && ip1_2 == ip2_2) {
|
|
||||||
vector_index_push_back(ip1_1, ip1_2, node);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
node.ip1 = ip1;
|
|
||||||
node.ip2 = ip_t(ip1, 255);
|
|
||||||
vector_index_push_back(ip1_1, ip1_2, node);
|
|
||||||
|
|
||||||
node.ip1 = ip_t(ip2, 0);
|
|
||||||
node.ip2 = ip2;
|
|
||||||
vector_index_push_back(ip2_1, ip2_2, node);
|
|
||||||
|
|
||||||
for (;;) {
|
|
||||||
++ip1_2;
|
|
||||||
if (ip1_2 == 256) {
|
|
||||||
++ip1_1;
|
|
||||||
ip1_2 = 0;
|
|
||||||
}
|
|
||||||
if (ip1_1 == ip2_1 && ip1_2 == ip2_2)
|
|
||||||
break;
|
|
||||||
ip1.p[0] = ip1_1;
|
|
||||||
ip1.p[1] = ip1_2;
|
|
||||||
node.ip1 = ip_t(ip1, 0);
|
|
||||||
node.ip2 = ip_t(ip1, 255);
|
|
||||||
vector_index_push_back(ip1_1, ip1_2, node);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
void make_t::handle_input_help(char *buf) {
|
|
||||||
// 去掉多余的空
|
|
||||||
unsigned int buf_len = strlen(buf);
|
|
||||||
while (buf_len > 0 && isspace(buf[buf_len - 1]))
|
|
||||||
--buf_len;
|
|
||||||
if (buf_len == 0)
|
|
||||||
return;
|
|
||||||
buf[buf_len] = '\0';
|
|
||||||
|
|
||||||
node_t node(buf);
|
|
||||||
|
|
||||||
if (node.ip1 < next_ip) {
|
|
||||||
log_exit("ip 未排序: " + node.ip1.to_string() + ", " +
|
|
||||||
next_ip.to_string());
|
|
||||||
}
|
|
||||||
|
|
||||||
next_ip = node.ip2 + 1;
|
|
||||||
|
|
||||||
if (region.find(node.region) == region.end()) {
|
|
||||||
region[node.region] = region_index;
|
|
||||||
region_index += node.region.size();
|
|
||||||
}
|
|
||||||
|
|
||||||
vector_index_push_back(node);
|
|
||||||
}
|
|
||||||
|
|
||||||
void make_t::handle_input(const std::string &file_name) {
|
|
||||||
FILE *src = fopen(file_name.data(), "r");
|
|
||||||
if (src == NULL)
|
|
||||||
log_exit("can't open " + file_name);
|
|
||||||
|
|
||||||
char buf[1024];
|
|
||||||
while (fgets(buf, sizeof(buf), src) != NULL)
|
|
||||||
handle_input_help(buf);
|
|
||||||
fclose(src);
|
|
||||||
}
|
|
||||||
|
|
||||||
void make_t::handle_header() {
|
|
||||||
char buf[length_header];
|
|
||||||
memset(buf, 0, length_header);
|
|
||||||
write_ushort(3, buf); // 版本号
|
|
||||||
write_ushort(1, buf + 2); // 缓存策略
|
|
||||||
write_uint(time(NULL), buf + 4); // 时间
|
|
||||||
// 索引
|
|
||||||
unsigned int content_left = length_header + length_vector;
|
|
||||||
for (auto &d : region)
|
|
||||||
content_left += d.first.size();
|
|
||||||
unsigned int content_right = content_left;
|
|
||||||
|
|
||||||
for (int i = 0; i < 256; ++i)
|
|
||||||
for (int j = 0; j < 256; ++j)
|
|
||||||
content_right += vector_index[i][j].size() * content_size;
|
|
||||||
content_right -= content_size;
|
|
||||||
write_uint(content_left, buf + 8);
|
|
||||||
write_uint(content_right, buf + 12);
|
|
||||||
write_ushort(ip_version, buf + 16); // IP
|
|
||||||
write_ushort(4, buf + 18); // 指针数
|
|
||||||
|
|
||||||
write_string(buf, length_header, db);
|
|
||||||
}
|
|
||||||
|
|
||||||
void make_t::handle_vector_index() {
|
|
||||||
unsigned index = length_header + length_vector;
|
|
||||||
for (auto &d : region)
|
|
||||||
index += d.first.size();
|
|
||||||
for (unsigned i = 0; i < 256; ++i)
|
|
||||||
for (unsigned j = 0; j < 256; ++j)
|
|
||||||
if (vector_index[i][j].size() == 0) {
|
|
||||||
write_uint(0, db);
|
|
||||||
write_uint(0, db);
|
|
||||||
} else {
|
|
||||||
write_uint(index, db);
|
|
||||||
index += content_size * vector_index[i][j].size();
|
|
||||||
write_uint(index, db);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
void make_t::handle_region() {
|
|
||||||
for (auto &d : region) {
|
|
||||||
fseek(db, d.second, SEEK_SET);
|
|
||||||
write_string(d.first.data(), d.first.size(), db);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
void make_t::handle_content() {
|
|
||||||
fseek(db, 0, SEEK_END);
|
|
||||||
for (unsigned i = 0; i < 256; ++i)
|
|
||||||
for (unsigned j = 0; j < 256; ++j)
|
|
||||||
for (auto d : vector_index[i][j]) {
|
|
||||||
write_string(d.first.data(), d.first.size(), db);
|
|
||||||
write_ushort(d.second.size(), db);
|
|
||||||
write_uint(region[d.second], db);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
make_t::make_t(const string &src, const string &dst, int version)
|
|
||||||
: region_index(length_vector + length_header) {
|
|
||||||
unsigned long long tv1 = get_time();
|
|
||||||
|
|
||||||
init_xdb(version);
|
|
||||||
|
|
||||||
handle_input(src);
|
|
||||||
|
|
||||||
db = fopen(dst.data(), "w");
|
|
||||||
if (db == NULL)
|
|
||||||
log_exit("can't open " + dst);
|
|
||||||
|
|
||||||
handle_header();
|
|
||||||
handle_vector_index();
|
|
||||||
handle_region();
|
|
||||||
handle_content();
|
|
||||||
|
|
||||||
fclose(db);
|
|
||||||
|
|
||||||
unsigned long long tv2 = get_time();
|
|
||||||
printf("took: %.2fs\n", (tv2 - tv1) * 1.0 / 1000 / 1000);
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
@ -1,35 +0,0 @@
|
||||||
#ifndef MAKE_H
|
|
||||||
#define MAKE_H
|
|
||||||
|
|
||||||
#include "ip.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
class make_t {
|
|
||||||
public:
|
|
||||||
make_t(const string &src, const string &dst, int version);
|
|
||||||
|
|
||||||
private:
|
|
||||||
void vector_index_push_back(int row, int col, const node_t &node);
|
|
||||||
void vector_index_push_back(node_t &node);
|
|
||||||
void handle_input_help(char buf[]);
|
|
||||||
void handle_input(const std::string &file_name);
|
|
||||||
|
|
||||||
void handle_header();
|
|
||||||
void handle_vector_index();
|
|
||||||
void handle_region();
|
|
||||||
void handle_content();
|
|
||||||
|
|
||||||
FILE *db = NULL;
|
|
||||||
|
|
||||||
std::vector<std::pair<string, string>> vector_index[256][256];
|
|
||||||
|
|
||||||
std::unordered_map<string, unsigned> region;
|
|
||||||
|
|
||||||
unsigned region_index;
|
|
||||||
ip_t next_ip;
|
|
||||||
};
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
||||||
#endif
|
|
||||||
|
|
@ -1,137 +0,0 @@
|
||||||
|
|
||||||
#include "search.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
search_t::search_t(const string &file, int version, int p)
|
|
||||||
: db(fopen(file.data(), "r")), header(db), policy(p) {
|
|
||||||
init_xdb(version);
|
|
||||||
|
|
||||||
if (db == NULL)
|
|
||||||
log_exit("can't open " + file);
|
|
||||||
if (header.ip_version() != version)
|
|
||||||
log_exit("ip 版本不匹配");
|
|
||||||
|
|
||||||
if (policy != policy_file) {
|
|
||||||
read_bin(length_header, vector, length_vector, db);
|
|
||||||
if (policy == policy_content) {
|
|
||||||
fseek(db, 0, SEEK_END);
|
|
||||||
int size = ftell(db) - length_vector - length_header;
|
|
||||||
content = (char *)malloc(size);
|
|
||||||
read_bin(length_vector + length_header, content, size, db);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
search_t::~search_t() {
|
|
||||||
fclose(db);
|
|
||||||
if (policy == policy_content)
|
|
||||||
free(content);
|
|
||||||
}
|
|
||||||
|
|
||||||
int search_t::get_io_count() {
|
|
||||||
return io_count;
|
|
||||||
}
|
|
||||||
|
|
||||||
int search_t::get_cost_time() {
|
|
||||||
return cost_time;
|
|
||||||
}
|
|
||||||
|
|
||||||
char const *search_t::get_content_index_help(int index) {
|
|
||||||
if (policy != policy_file)
|
|
||||||
return vector + index;
|
|
||||||
|
|
||||||
++io_count;
|
|
||||||
static char v[8];
|
|
||||||
read_bin(length_header + index, v, sizeof(v), db);
|
|
||||||
return v;
|
|
||||||
}
|
|
||||||
|
|
||||||
void search_t::get_content_index(const ip_t &ip, int &left, int &right) {
|
|
||||||
int index = ((unsigned char)ip.p[0] * 256 + (unsigned char)ip.p[1]) * 8;
|
|
||||||
|
|
||||||
const char *p = get_content_index_help(index);
|
|
||||||
left = to_uint(p);
|
|
||||||
right = to_uint(p + 4);
|
|
||||||
}
|
|
||||||
|
|
||||||
char const *search_t::get_content_help(int index) {
|
|
||||||
if (policy == policy_content)
|
|
||||||
return content + index - length_header - length_vector;
|
|
||||||
++io_count;
|
|
||||||
static char v[16 + 16 + 2 + 4];
|
|
||||||
read_bin(index, v, content_size, db);
|
|
||||||
return v;
|
|
||||||
}
|
|
||||||
|
|
||||||
string search_t::get_region(int index, int len) {
|
|
||||||
if (policy == policy_content)
|
|
||||||
return string(content + index - length_header - length_vector, len);
|
|
||||||
++io_count;
|
|
||||||
char *p = (char *)malloc(sizeof(char) * len);
|
|
||||||
read_bin(index, p, len, db);
|
|
||||||
string res(p, len);
|
|
||||||
free(p);
|
|
||||||
return res;
|
|
||||||
}
|
|
||||||
|
|
||||||
void search_t::get_content(int index,
|
|
||||||
ip_t &ip_left,
|
|
||||||
ip_t &ip_right,
|
|
||||||
int ®ion_len,
|
|
||||||
int ®ion_index) {
|
|
||||||
const char *p = get_content_help(index);
|
|
||||||
|
|
||||||
ip_left.from_xdb(p);
|
|
||||||
ip_right.from_xdb(p + ip_size);
|
|
||||||
|
|
||||||
region_len = to_ushort(p + ip_size * 2);
|
|
||||||
region_index = to_uint(p + ip_size * 2 + 2);
|
|
||||||
}
|
|
||||||
|
|
||||||
string search_t::search(const ip_t &ip) {
|
|
||||||
io_count = 0;
|
|
||||||
|
|
||||||
int content_left, content_right;
|
|
||||||
get_content_index(ip, content_left, content_right);
|
|
||||||
|
|
||||||
if (content_left == 0 || content_right == 0)
|
|
||||||
return "";
|
|
||||||
|
|
||||||
ip_t ip_left, ip_right;
|
|
||||||
int region_len;
|
|
||||||
int region_index;
|
|
||||||
|
|
||||||
int left = 0;
|
|
||||||
int right = (content_right - content_left) / content_size;
|
|
||||||
|
|
||||||
for (;;) {
|
|
||||||
int mid = left + (right - left) / 2;
|
|
||||||
int mid_index = content_left + mid * content_size;
|
|
||||||
get_content(mid_index, ip_left, ip_right, region_len, region_index);
|
|
||||||
|
|
||||||
// ip ip_left ip_right
|
|
||||||
if (ip < ip_left)
|
|
||||||
right = mid - 1;
|
|
||||||
// ip_left ip_right ip
|
|
||||||
else if (ip_right < ip)
|
|
||||||
left = mid + 1;
|
|
||||||
else
|
|
||||||
return get_region(region_index, region_len);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
string search_t::search(const string &str) {
|
|
||||||
unsigned long long t1 = get_time();
|
|
||||||
|
|
||||||
ip_t ip;
|
|
||||||
if (ip.from_str(str) == false)
|
|
||||||
return "invalid ipv" + std::to_string(ip_version) + ": " + str;
|
|
||||||
string region = search(ip);
|
|
||||||
|
|
||||||
unsigned long long t2 = get_time();
|
|
||||||
cost_time = t2 - t1;
|
|
||||||
return region;
|
|
||||||
}
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
@ -1,49 +0,0 @@
|
||||||
#ifndef SEARCH_H
|
|
||||||
#define SEARCH_H
|
|
||||||
|
|
||||||
#include "header.h"
|
|
||||||
#include "ip.h"
|
|
||||||
|
|
||||||
namespace xdb {
|
|
||||||
|
|
||||||
class search_t {
|
|
||||||
protected:
|
|
||||||
FILE *db;
|
|
||||||
|
|
||||||
header_t header;
|
|
||||||
|
|
||||||
int policy;
|
|
||||||
|
|
||||||
int io_count;
|
|
||||||
int cost_time;
|
|
||||||
|
|
||||||
char vector[length_vector];
|
|
||||||
char *content;
|
|
||||||
|
|
||||||
public:
|
|
||||||
search_t(const string &file_name, int version, int policy);
|
|
||||||
virtual ~search_t();
|
|
||||||
|
|
||||||
int get_io_count();
|
|
||||||
int get_cost_time();
|
|
||||||
|
|
||||||
string search(const string &str);
|
|
||||||
|
|
||||||
protected:
|
|
||||||
string search(const ip_t &ip);
|
|
||||||
|
|
||||||
void get_content_index(const ip_t &ip1, int &left, int &right);
|
|
||||||
void get_content(int index,
|
|
||||||
ip_t &left,
|
|
||||||
ip_t &right,
|
|
||||||
int ®ion_len,
|
|
||||||
int ®ion_index);
|
|
||||||
|
|
||||||
char const *get_content_index_help(int index);
|
|
||||||
char const *get_content_help(int index);
|
|
||||||
string get_region(int index, int len);
|
|
||||||
};
|
|
||||||
|
|
||||||
} // namespace xdb
|
|
||||||
|
|
||||||
#endif
|
|
||||||
|
|
@ -1,31 +0,0 @@
|
||||||
|
|
||||||
#include "../src/bench.h"
|
|
||||||
|
|
||||||
std::map<int, std::string> prompt;
|
|
||||||
|
|
||||||
void test_ipv4(int policy) {
|
|
||||||
std::cout << "测试 IPv4, " << prompt[policy];
|
|
||||||
xdb::bench_t("../../data/ip2region_v4.xdb", xdb::ipv4, policy)
|
|
||||||
.test("../../data/ipv4_source.txt");
|
|
||||||
}
|
|
||||||
|
|
||||||
void test_ipv6(int policy) {
|
|
||||||
std::cout << "测试 IPv6, " << prompt[policy];
|
|
||||||
xdb::bench_t("../../data/ip2region_v6.xdb", xdb::ipv6, policy)
|
|
||||||
.test("../../data/ipv6_source.txt");
|
|
||||||
}
|
|
||||||
|
|
||||||
int main() {
|
|
||||||
prompt[xdb::policy_file] = " 不缓存, ";
|
|
||||||
prompt[xdb::policy_vector] = "部分缓存, ";
|
|
||||||
prompt[xdb::policy_content] = "全部缓存, ";
|
|
||||||
|
|
||||||
test_ipv4(xdb::policy_file);
|
|
||||||
test_ipv4(xdb::policy_vector);
|
|
||||||
test_ipv4(xdb::policy_content);
|
|
||||||
|
|
||||||
test_ipv6(xdb::policy_file);
|
|
||||||
test_ipv6(xdb::policy_vector);
|
|
||||||
test_ipv6(xdb::policy_content);
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
@ -1,9 +0,0 @@
|
||||||
|
|
||||||
#include "../src/edit.h"
|
|
||||||
|
|
||||||
int main() {
|
|
||||||
std::string file_name_old = "../../data/ipv4_source.txt";
|
|
||||||
std::string file_name_new = "./1.txt";
|
|
||||||
xdb::edit_t xdb(file_name_old, file_name_new, xdb::ipv4);
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
@ -1,9 +0,0 @@
|
||||||
|
|
||||||
#include "../src/edit.h"
|
|
||||||
|
|
||||||
int main() {
|
|
||||||
std::string file_name_old = "../../data/ipv6_source.txt";
|
|
||||||
std::string file_name_new = "./1.txt";
|
|
||||||
xdb::edit_t xdb(file_name_old, file_name_new, xdb::ipv6);
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
@ -1,30 +0,0 @@
|
||||||
|
|
||||||
#include "../src/header.h"
|
|
||||||
|
|
||||||
void test(const std::string& prompt, const std::string& file_name) {
|
|
||||||
std::cout << prompt << std::endl;
|
|
||||||
|
|
||||||
xdb::header_t head(fopen(file_name.data(), "r"));
|
|
||||||
|
|
||||||
std::cout << "版本号: " << head.version() << std::endl;
|
|
||||||
std::cout << "缓存策略: " << head.index_policy() << std::endl;
|
|
||||||
|
|
||||||
time_t rawtime = head.create_at();
|
|
||||||
struct tm* info = localtime(&rawtime);
|
|
||||||
char buf[80];
|
|
||||||
strftime(buf, 80, "%Y-%m-%d %H:%M:%S", info);
|
|
||||||
|
|
||||||
std::cout << "文件生成时间: " << buf << std::endl;
|
|
||||||
std::cout << "索引起始地址: " << head.index_start() << std::endl;
|
|
||||||
std::cout << "索引结束地址: " << head.index_end() << std::endl;
|
|
||||||
std::cout << "IP版本: " << head.ip_version() << std::endl;
|
|
||||||
std::cout << "指针字节数: " << head.ptr() << std::endl;
|
|
||||||
|
|
||||||
std::cout << std::endl;
|
|
||||||
}
|
|
||||||
|
|
||||||
int main() {
|
|
||||||
test("测试 IPv4", "../../data/ip2region_v4.xdb");
|
|
||||||
test("测试 IPv6", "../../data/ip2region_v6.xdb");
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
@ -1,26 +0,0 @@
|
||||||
|
|
||||||
#include "../src/make.h"
|
|
||||||
|
|
||||||
void test(const std::string& prompt,
|
|
||||||
const std::string& filename_xdb,
|
|
||||||
const std::string& filename_src,
|
|
||||||
int version
|
|
||||||
|
|
||||||
) {
|
|
||||||
std::cout << prompt;
|
|
||||||
xdb::make_t(filename_xdb, filename_src, version);
|
|
||||||
}
|
|
||||||
|
|
||||||
int main() {
|
|
||||||
test("生成 ipv4 的 xdb 文件, ",
|
|
||||||
"../../data/ipv4_source.txt",
|
|
||||||
"./ip2region_v4.xdb",
|
|
||||||
xdb::ipv4);
|
|
||||||
|
|
||||||
test("生成 ipv6 的 xdb 文件, ",
|
|
||||||
"../../data/ipv6_source.txt",
|
|
||||||
"./ip2region_v6.xdb",
|
|
||||||
xdb::ipv6);
|
|
||||||
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
@ -1,46 +0,0 @@
|
||||||
|
|
||||||
#include "../src/search.h"
|
|
||||||
|
|
||||||
std::map<int, std::string> prompt;
|
|
||||||
|
|
||||||
void test(xdb::search_t& s, const std::string& ip, const std::string& region) {
|
|
||||||
if (s.search(ip) != region)
|
|
||||||
xdb::log_exit("测试失败, ip " + ip + ", region " + region);
|
|
||||||
}
|
|
||||||
|
|
||||||
void test_ipv4(int policy) {
|
|
||||||
std::cout << "测试 IPv4 " << prompt[policy];
|
|
||||||
|
|
||||||
xdb::search_t s("../../data/ip2region_v4.xdb", xdb::ipv4, policy);
|
|
||||||
test(s, "0.0.0.0", "Reserved|Reserved|Reserved|0|0");
|
|
||||||
test(s, "1.2.3.4", "Australia|Queensland|Brisbane|0|AU");
|
|
||||||
|
|
||||||
std::cout << " 成功" << std::endl;
|
|
||||||
}
|
|
||||||
|
|
||||||
void test_ipv6(int policy) {
|
|
||||||
std::cout << "测试 IPv6 " << prompt[policy];
|
|
||||||
|
|
||||||
xdb::search_t s("../../data/ip2region_v6.xdb", xdb::ipv6, policy);
|
|
||||||
test(s, "::1", "");
|
|
||||||
test(s, "2001:200:124::", "Japan|Tokyo|Asagaya-minami|WIDE Project|JP");
|
|
||||||
test(s, "2001:200:124::", "Japan|Tokyo|Asagaya-minami|WIDE Project|JP");
|
|
||||||
test(s, "240e:3b7:3273:51d0:cd38:8ae1:e3c0:b708", "中国|广东省|深圳市|电信|CN");
|
|
||||||
|
|
||||||
std::cout << " 成功" << std::endl;
|
|
||||||
}
|
|
||||||
|
|
||||||
int main() {
|
|
||||||
prompt[xdb::policy_file] = " 不缓存:";
|
|
||||||
prompt[xdb::policy_vector] = "部分缓存:";
|
|
||||||
prompt[xdb::policy_content] = "全部缓存:";
|
|
||||||
|
|
||||||
test_ipv4(xdb::policy_file);
|
|
||||||
test_ipv4(xdb::policy_vector);
|
|
||||||
test_ipv4(xdb::policy_content);
|
|
||||||
|
|
||||||
test_ipv6(xdb::policy_file);
|
|
||||||
test_ipv6(xdb::policy_vector);
|
|
||||||
test_ipv6(xdb::policy_content);
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
@ -0,0 +1,135 @@
|
||||||
|
|
||||||
|
#include "xdb_bench.h"
|
||||||
|
|
||||||
|
#include <arpa/inet.h>
|
||||||
|
#include <stdio.h>
|
||||||
|
#include <string.h>
|
||||||
|
#include <sys/time.h>
|
||||||
|
|
||||||
|
#include <iostream>
|
||||||
|
#include <vector>
|
||||||
|
|
||||||
|
static void log_exit(const std::string &msg) {
|
||||||
|
std::cout << msg << std::endl;
|
||||||
|
exit(-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
static unsigned long long get_time() {
|
||||||
|
struct timeval tv1;
|
||||||
|
gettimeofday(&tv1, NULL);
|
||||||
|
return (unsigned long long)tv1.tv_sec * 1000 * 1000 + tv1.tv_usec;
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool ip2uint(const char *buf, unsigned int &ip) {
|
||||||
|
struct in_addr addr;
|
||||||
|
if (inet_pton(AF_INET, buf, &addr) == 0)
|
||||||
|
return false;
|
||||||
|
// 网络字节序为大端存储, 在此转换为小端存储
|
||||||
|
ip = (((addr.s_addr >> 0) & 0xFF) << 24) |
|
||||||
|
(((addr.s_addr >> 8) & 0xFF) << 16) |
|
||||||
|
(((addr.s_addr >> 16) & 0xFF) << 8) |
|
||||||
|
(((addr.s_addr >> 24) & 0xFF) << 0);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
static std::string uint2ip(unsigned int ip) {
|
||||||
|
char buf[16];
|
||||||
|
snprintf(buf,
|
||||||
|
sizeof(buf),
|
||||||
|
"%d.%d.%d.%d",
|
||||||
|
(ip >> 24) & 0xFF,
|
||||||
|
(ip >> 16) & 0xFF,
|
||||||
|
(ip >> 8) & 0xFF,
|
||||||
|
ip & 0xFF);
|
||||||
|
return std::string(buf);
|
||||||
|
}
|
||||||
|
|
||||||
|
xdb_bench_t::xdb_bench_t(const std::string &file_name) : xdb_search(file_name) {
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_bench_t::init_file() {
|
||||||
|
xdb_search.init_file();
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_bench_t::init_vector_index() {
|
||||||
|
xdb_search.init_vector_index();
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_bench_t::init_content() {
|
||||||
|
xdb_search.init_content();
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_bench_t::bench_test_one(unsigned int ip_uint, const char *region) {
|
||||||
|
if (xdb_search.search(uint2ip(ip_uint)) != region)
|
||||||
|
log_exit("failed: " + uint2ip(ip_uint));
|
||||||
|
sum_io_count += xdb_search.get_io_count();
|
||||||
|
sum_cost_time += xdb_search.get_cost_time();
|
||||||
|
sum_count++;
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_bench_t::bench_test_line(char *buf) {
|
||||||
|
size_t buf_len = strlen(buf);
|
||||||
|
if (buf_len == 0)
|
||||||
|
return;
|
||||||
|
buf[buf_len - 1] = '\0'; // 去掉换行符
|
||||||
|
|
||||||
|
char *pos1 = strchr(buf, '|');
|
||||||
|
|
||||||
|
if (pos1 == NULL)
|
||||||
|
log_exit("invalid data: " + std::string(buf));
|
||||||
|
char *pos2 = strchr(pos1 + 1, '|');
|
||||||
|
if (pos2 == NULL)
|
||||||
|
log_exit("invalid data: " + std::string(buf));
|
||||||
|
*pos1 = '\0';
|
||||||
|
*pos2 = '\0';
|
||||||
|
|
||||||
|
unsigned int ip1, ip2;
|
||||||
|
if (!ip2uint(buf, ip1) || !ip2uint(pos1 + 1, ip2) || ip1 > ip2) {
|
||||||
|
*pos1 = *pos2 = '|';
|
||||||
|
log_exit(std::string("invalid data: ") + buf);
|
||||||
|
}
|
||||||
|
|
||||||
|
const char *region = pos2 + 1;
|
||||||
|
|
||||||
|
unsigned int ip_mid = ip1 + (ip2 - ip1) / 2;
|
||||||
|
std::vector<unsigned int> ip_vec;
|
||||||
|
ip_vec.push_back(ip1);
|
||||||
|
ip_vec.push_back(ip1 + (ip_mid - ip1) / 2);
|
||||||
|
ip_vec.push_back(ip_mid);
|
||||||
|
ip_vec.push_back(ip_mid + (ip2 - ip_mid) / 2);
|
||||||
|
ip_vec.push_back(ip2);
|
||||||
|
|
||||||
|
for (auto &d : ip_vec)
|
||||||
|
bench_test_one(d, region);
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_bench_t::bench_test_file(const std::string &file_name) {
|
||||||
|
FILE *f = fopen(file_name.data(), "r");
|
||||||
|
if (f == NULL)
|
||||||
|
log_exit("can't open " + file_name);
|
||||||
|
char buf[1024];
|
||||||
|
while (fgets(buf, sizeof(buf), f) != NULL)
|
||||||
|
bench_test_line(buf);
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_bench_t::bench(const std::string &file_name) {
|
||||||
|
sum_io_count = 0;
|
||||||
|
sum_cost_time = 0;
|
||||||
|
sum_count = 0;
|
||||||
|
|
||||||
|
unsigned long long tv1 = get_time();
|
||||||
|
bench_test_file(file_name);
|
||||||
|
unsigned long long tv2 = get_time();
|
||||||
|
|
||||||
|
double took = (tv2 - tv1) * 1.0 / 1000 / 1000;
|
||||||
|
double cost = sum_cost_time * 1.0 / sum_count;
|
||||||
|
|
||||||
|
printf(
|
||||||
|
"total: %llu, took: %.2f s, cost: %.2f μs/op, io "
|
||||||
|
"count: "
|
||||||
|
"%llu\n",
|
||||||
|
sum_count,
|
||||||
|
took,
|
||||||
|
cost,
|
||||||
|
sum_io_count);
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,28 @@
|
||||||
|
#ifndef XDB_BENCH_H
|
||||||
|
#define XDB_BENCH_H
|
||||||
|
|
||||||
|
#include "xdb_search.h"
|
||||||
|
|
||||||
|
class xdb_bench_t {
|
||||||
|
public:
|
||||||
|
xdb_bench_t(const std::string &file_name);
|
||||||
|
|
||||||
|
void init_file();
|
||||||
|
void init_vector_index();
|
||||||
|
void init_content();
|
||||||
|
|
||||||
|
void bench(const std::string &file_name);
|
||||||
|
|
||||||
|
private:
|
||||||
|
void bench_test_one(unsigned int ip_uint, const char *region);
|
||||||
|
void bench_test_line(char *buf);
|
||||||
|
void bench_test_file(const std::string &file_name);
|
||||||
|
|
||||||
|
xdb_search_t xdb_search;
|
||||||
|
|
||||||
|
unsigned long long sum_io_count;
|
||||||
|
unsigned long long sum_cost_time;
|
||||||
|
unsigned long long sum_count;
|
||||||
|
};
|
||||||
|
|
||||||
|
#endif
|
||||||
|
|
@ -0,0 +1,70 @@
|
||||||
|
|
||||||
|
#include "xdb_bench.h"
|
||||||
|
|
||||||
|
#include <getopt.h>
|
||||||
|
|
||||||
|
#include <iostream>
|
||||||
|
|
||||||
|
void print_help(int argc, char* argv[]) {
|
||||||
|
printf("./xdb_bench [command options]\n");
|
||||||
|
printf("options:\n");
|
||||||
|
printf(" --db string ip2region binary xdb file path\n");
|
||||||
|
printf(" --src string source ip text file path\n");
|
||||||
|
printf(
|
||||||
|
" --cache-policy string cache policy: "
|
||||||
|
"file/vector_index/content\n");
|
||||||
|
printf(" --help print help\n");
|
||||||
|
exit(-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(int argc, char* argv[]) {
|
||||||
|
struct option long_options[] = {
|
||||||
|
{"db", required_argument, 0, 'd'},
|
||||||
|
{"cache-policy", required_argument, 0, 't'},
|
||||||
|
{"src", required_argument, 0, 's'},
|
||||||
|
{"help", no_argument, 0, 'h'},
|
||||||
|
{0, 0, 0, 0 }
|
||||||
|
};
|
||||||
|
|
||||||
|
std::string db_file_name = "../../data/ip2region.xdb";
|
||||||
|
std::string src_file_name = "../../data/ip.merge.txt";
|
||||||
|
std::string cache_policy = "vector_index";
|
||||||
|
|
||||||
|
while (1) {
|
||||||
|
int c = getopt_long(argc, argv, "", long_options, NULL);
|
||||||
|
if (c == -1)
|
||||||
|
break;
|
||||||
|
switch (c) {
|
||||||
|
case 'd':
|
||||||
|
db_file_name = optarg;
|
||||||
|
break;
|
||||||
|
case 'h':
|
||||||
|
print_help(argc, argv);
|
||||||
|
break;
|
||||||
|
case 't':
|
||||||
|
cache_policy = optarg;
|
||||||
|
break;
|
||||||
|
case 's':
|
||||||
|
src_file_name = optarg;
|
||||||
|
break;
|
||||||
|
case '?':
|
||||||
|
exit(-1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
xdb_bench_t xdb(db_file_name);
|
||||||
|
|
||||||
|
if (cache_policy == "content")
|
||||||
|
xdb.init_content();
|
||||||
|
else if (cache_policy == "vector_index")
|
||||||
|
xdb.init_vector_index();
|
||||||
|
else if (cache_policy == "file")
|
||||||
|
xdb.init_file();
|
||||||
|
else {
|
||||||
|
std::cout << "invalid cache policy: " << cache_policy << std::endl;
|
||||||
|
exit(-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
xdb.bench(src_file_name);
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,190 @@
|
||||||
|
|
||||||
|
#include "xdb_search.h"
|
||||||
|
|
||||||
|
#include <arpa/inet.h>
|
||||||
|
#include <sys/time.h>
|
||||||
|
|
||||||
|
#include <iostream>
|
||||||
|
|
||||||
|
static void log_exit(const std::string &msg) {
|
||||||
|
std::cout << msg << std::endl;
|
||||||
|
exit(-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
static unsigned long long get_time() {
|
||||||
|
struct timeval tv1;
|
||||||
|
gettimeofday(&tv1, NULL);
|
||||||
|
return (unsigned long long)tv1.tv_sec * 1000 * 1000 + tv1.tv_usec;
|
||||||
|
}
|
||||||
|
|
||||||
|
static void read_bin(int index, char *buf, size_t len, FILE *db) {
|
||||||
|
fseek(db, index, SEEK_SET);
|
||||||
|
if (fread(buf, 1, len, db) != len)
|
||||||
|
log_exit(__func__);
|
||||||
|
}
|
||||||
|
|
||||||
|
static unsigned int read_uint(const char *buf) {
|
||||||
|
return ((buf[0]) & 0x000000FF) | ((buf[1] << 8) & 0x0000FF00) |
|
||||||
|
((buf[2] << 16) & 0x00FF0000) | ((buf[3] << 24) & 0xFF000000);
|
||||||
|
}
|
||||||
|
|
||||||
|
static unsigned short read_ushort(const char *buf) {
|
||||||
|
return ((buf[0]) & 0x000000FF) | ((buf[1] << 8) & 0x0000FF00);
|
||||||
|
}
|
||||||
|
|
||||||
|
static bool ip2uint(const char *buf, unsigned int &ip) {
|
||||||
|
struct in_addr addr;
|
||||||
|
if (inet_pton(AF_INET, buf, &addr) == 0)
|
||||||
|
return false;
|
||||||
|
// 网络字节序为大端存储, 在此转换为小端存储
|
||||||
|
ip = (((addr.s_addr >> 0) & 0xFF) << 24) |
|
||||||
|
(((addr.s_addr >> 8) & 0xFF) << 16) |
|
||||||
|
(((addr.s_addr >> 16) & 0xFF) << 8) |
|
||||||
|
(((addr.s_addr >> 24) & 0xFF) << 0);
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_search_t::get_content_index(unsigned int ip,
|
||||||
|
unsigned int &left,
|
||||||
|
unsigned int &right) {
|
||||||
|
unsigned int ip_1 = (ip >> 24) & 0xFF;
|
||||||
|
unsigned int ip_2 = (ip >> 16) & 0xFF;
|
||||||
|
unsigned int index = (ip_1 * vector_index_cols + ip_2) * vector_index_size;
|
||||||
|
|
||||||
|
if (content != NULL) {
|
||||||
|
left = read_uint(content + index + header_length);
|
||||||
|
right = read_uint(content + index + header_length + 4);
|
||||||
|
} else if (vector_index != NULL) {
|
||||||
|
left = read_uint(vector_index + index);
|
||||||
|
right = read_uint(vector_index + index + 4);
|
||||||
|
} else {
|
||||||
|
++io_count;
|
||||||
|
char buf[8];
|
||||||
|
read_bin(header_length + index, buf, sizeof(buf), db);
|
||||||
|
left = read_uint(buf);
|
||||||
|
right = read_uint(buf + 4);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_search_t::get_content(unsigned int index,
|
||||||
|
unsigned int &ip_left,
|
||||||
|
unsigned int &ip_right,
|
||||||
|
unsigned short ®ion_len,
|
||||||
|
unsigned int ®ion_index) {
|
||||||
|
char buf[segment_index_size]; // 4 + 4 + 2 + 4
|
||||||
|
const char *p;
|
||||||
|
|
||||||
|
if (content != NULL) {
|
||||||
|
p = content + index;
|
||||||
|
} else {
|
||||||
|
++io_count;
|
||||||
|
read_bin(index, buf, sizeof(buf), db);
|
||||||
|
p = buf;
|
||||||
|
}
|
||||||
|
ip_left = read_uint(p);
|
||||||
|
ip_right = read_uint(p + 4);
|
||||||
|
region_len = read_ushort(p + 8);
|
||||||
|
region_index = read_uint(p + 10);
|
||||||
|
}
|
||||||
|
|
||||||
|
std::string xdb_search_t::get_region(unsigned int index, unsigned short len) {
|
||||||
|
if (content != NULL) {
|
||||||
|
return std::string(content + index, len);
|
||||||
|
} else {
|
||||||
|
++io_count;
|
||||||
|
char *buf = (char *)malloc(sizeof(char) * len);
|
||||||
|
read_bin(index, buf, len, db);
|
||||||
|
std::string res(buf, len);
|
||||||
|
free(buf);
|
||||||
|
return res;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
xdb_search_t::xdb_search_t(const std::string &file_name) {
|
||||||
|
db = fopen(file_name.data(), "r");
|
||||||
|
vector_index = NULL;
|
||||||
|
content = NULL;
|
||||||
|
|
||||||
|
if (db == NULL)
|
||||||
|
log_exit("can't open " + file_name);
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_search_t::init_file() {
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_search_t::init_vector_index() {
|
||||||
|
vector_index = (char *)malloc(vector_index_length);
|
||||||
|
read_bin(header_length, vector_index, vector_index_length, db);
|
||||||
|
}
|
||||||
|
|
||||||
|
void xdb_search_t::init_content() {
|
||||||
|
fseek(db, 0, SEEK_END);
|
||||||
|
unsigned int size = ftell(db);
|
||||||
|
content = (char *)malloc(size);
|
||||||
|
read_bin(0, content, size, db);
|
||||||
|
}
|
||||||
|
|
||||||
|
xdb_search_t::~xdb_search_t() {
|
||||||
|
if (db != NULL) {
|
||||||
|
fclose(db);
|
||||||
|
db = NULL;
|
||||||
|
}
|
||||||
|
if (vector_index != NULL) {
|
||||||
|
free(vector_index);
|
||||||
|
vector_index = NULL;
|
||||||
|
}
|
||||||
|
if (content != NULL) {
|
||||||
|
free(content);
|
||||||
|
content = NULL;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
unsigned long long xdb_search_t::get_io_count() {
|
||||||
|
return io_count;
|
||||||
|
}
|
||||||
|
|
||||||
|
unsigned long long xdb_search_t::get_cost_time() {
|
||||||
|
return cost_time;
|
||||||
|
}
|
||||||
|
|
||||||
|
std::string xdb_search_t::search(const std::string &ip_str) {
|
||||||
|
unsigned long long t1 = get_time();
|
||||||
|
|
||||||
|
unsigned int ip_uint;
|
||||||
|
if (!ip2uint(ip_str.data(), ip_uint))
|
||||||
|
return "invalid ip: " + ip_str;
|
||||||
|
std::string region = search(ip_uint);
|
||||||
|
|
||||||
|
unsigned long long t2 = get_time();
|
||||||
|
cost_time = t2 - t1;
|
||||||
|
return region;
|
||||||
|
}
|
||||||
|
|
||||||
|
std::string xdb_search_t::search(unsigned int ip_uint) {
|
||||||
|
io_count = 0;
|
||||||
|
|
||||||
|
unsigned int content_index_left, content_index_right;
|
||||||
|
get_content_index(ip_uint, content_index_left, content_index_right);
|
||||||
|
|
||||||
|
unsigned int left, right, mid;
|
||||||
|
unsigned int ip_left, ip_right;
|
||||||
|
unsigned short region_len;
|
||||||
|
unsigned int region_index;
|
||||||
|
unsigned int mid_index;
|
||||||
|
|
||||||
|
left = 0;
|
||||||
|
right = (content_index_right - content_index_left) / segment_index_size;
|
||||||
|
|
||||||
|
for (;;) {
|
||||||
|
mid = left + (right - left) / 2;
|
||||||
|
mid_index = content_index_left + mid * segment_index_size;
|
||||||
|
get_content(mid_index, ip_left, ip_right, region_len, region_index);
|
||||||
|
|
||||||
|
if (ip_left > ip_uint)
|
||||||
|
right = mid - 1;
|
||||||
|
else if (ip_right < ip_uint)
|
||||||
|
left = mid + 1;
|
||||||
|
else
|
||||||
|
return get_region(region_index, region_len);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -0,0 +1,50 @@
|
||||||
|
#ifndef XDB_SEARCH_H
|
||||||
|
#define XDB_SEARCH_H
|
||||||
|
|
||||||
|
#include <string>
|
||||||
|
|
||||||
|
class xdb_search_t {
|
||||||
|
public:
|
||||||
|
xdb_search_t(const std::string &file_name);
|
||||||
|
~xdb_search_t();
|
||||||
|
|
||||||
|
void init_file();
|
||||||
|
void init_vector_index();
|
||||||
|
void init_content();
|
||||||
|
|
||||||
|
unsigned long long get_io_count();
|
||||||
|
unsigned long long get_cost_time();
|
||||||
|
|
||||||
|
std::string search(const std::string &ip);
|
||||||
|
|
||||||
|
private:
|
||||||
|
void get_content_index(unsigned int ip,
|
||||||
|
unsigned int &left,
|
||||||
|
unsigned int &right);
|
||||||
|
|
||||||
|
void get_content(unsigned int index,
|
||||||
|
unsigned int &ip_left,
|
||||||
|
unsigned int &ip_right,
|
||||||
|
unsigned short ®ion_len,
|
||||||
|
unsigned int ®ion_index);
|
||||||
|
|
||||||
|
std::string get_region(unsigned int index, unsigned short len);
|
||||||
|
|
||||||
|
std::string search(unsigned int ip_uint);
|
||||||
|
|
||||||
|
FILE *db;
|
||||||
|
char *vector_index;
|
||||||
|
char *content;
|
||||||
|
unsigned long long io_count;
|
||||||
|
unsigned long long cost_time;
|
||||||
|
|
||||||
|
static constexpr int header_length = 256;
|
||||||
|
static constexpr int vector_index_rows = 256;
|
||||||
|
static constexpr int vector_index_cols = 256;
|
||||||
|
static constexpr int vector_index_size = 8;
|
||||||
|
static constexpr int vector_index_length =
|
||||||
|
vector_index_rows * vector_index_cols * vector_index_size;
|
||||||
|
static constexpr int segment_index_size = 14;
|
||||||
|
};
|
||||||
|
|
||||||
|
#endif
|
||||||
|
|
@ -0,0 +1,73 @@
|
||||||
|
|
||||||
|
#include "xdb_search.h"
|
||||||
|
|
||||||
|
#include <getopt.h>
|
||||||
|
|
||||||
|
#include <iostream>
|
||||||
|
|
||||||
|
void print_help(int argc, char* argv[]) {
|
||||||
|
printf("./xdb_search [command options]\n");
|
||||||
|
printf("options:\n");
|
||||||
|
printf(" --db string ip2region binary xdb file path\n");
|
||||||
|
printf(
|
||||||
|
" --cache-policy string cache policy: "
|
||||||
|
"file/vector_index/content\n");
|
||||||
|
printf(" --help print help\n");
|
||||||
|
exit(-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
int main(int argc, char* argv[]) {
|
||||||
|
struct option long_options[] = {
|
||||||
|
{"db", required_argument, 0, 'd'},
|
||||||
|
{"cache-policy", required_argument, 0, 't'},
|
||||||
|
{"help", no_argument, 0, 'h'},
|
||||||
|
{0, 0, 0, 0 }
|
||||||
|
};
|
||||||
|
|
||||||
|
std::string db_file_name = "../../data/ip2region.xdb";
|
||||||
|
std::string cache_policy = "vector_index";
|
||||||
|
|
||||||
|
while (1) {
|
||||||
|
int c = getopt_long(argc, argv, "", long_options, NULL);
|
||||||
|
if (c == -1)
|
||||||
|
break;
|
||||||
|
switch (c) {
|
||||||
|
case 'd':
|
||||||
|
db_file_name = optarg;
|
||||||
|
break;
|
||||||
|
case 'h':
|
||||||
|
print_help(argc, argv);
|
||||||
|
break;
|
||||||
|
case 't':
|
||||||
|
cache_policy = optarg;
|
||||||
|
break;
|
||||||
|
case '?':
|
||||||
|
exit(-1);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
xdb_search_t xdb(db_file_name);
|
||||||
|
|
||||||
|
if (cache_policy == "content")
|
||||||
|
xdb.init_content();
|
||||||
|
else if (cache_policy == "vector_index")
|
||||||
|
xdb.init_vector_index();
|
||||||
|
else if (cache_policy == "file")
|
||||||
|
xdb.init_file();
|
||||||
|
else {
|
||||||
|
std::cout << "invalid cache policy: " << cache_policy << std::endl;
|
||||||
|
exit(-1);
|
||||||
|
}
|
||||||
|
|
||||||
|
std::string ip;
|
||||||
|
for (;;) {
|
||||||
|
std::cout << "ip2region>> ";
|
||||||
|
std::getline(std::cin, ip);
|
||||||
|
if (ip.empty())
|
||||||
|
continue;
|
||||||
|
if (ip == "exit" || ip == "quit")
|
||||||
|
break;
|
||||||
|
std::cout << xdb.search(ip) << std::endl;
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
root = true
|
|
||||||
|
|
||||||
# To learn more about .editorconfig see https://aka.ms/editorconfigdocs
|
|
||||||
[*]
|
|
||||||
charset = utf-8
|
|
||||||
indent_style = space
|
|
||||||
trim_trailing_whitespace = true
|
|
||||||
insert_final_newline = true
|
|
||||||
spelling_exclusion_path = .\exclusion.dic
|
|
||||||
|
|
||||||
[*.csproj]
|
|
||||||
charset = utf-8
|
|
||||||
insert_final_newline = true
|
|
||||||
|
|
||||||
[*.{xml,config,csproj,nuspec,props,resx,targets,yml,tasks,json}]
|
|
||||||
indent_size = 2
|
|
||||||
|
|
||||||
[*.sh]
|
|
||||||
end_of_line = lf
|
|
||||||
|
|
||||||
[*.cs]
|
|
||||||
csharp_style_namespace_declarations = file_scoped:silent
|
|
||||||
|
|
||||||
[*.cs]
|
|
||||||
file_header_template = Copyright 2025 The Ip2Region Authors. All rights reserved.\nUse of this source code is governed by a Apache2.0-style\nlicense that can be found in the LICENSE file.\n@Author Alan <lzh.shap@gmail.com>\n@Date 2023/07/25\nUpdated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
@ -2,10 +2,7 @@
|
||||||
|
|
||||||
All notable changes to this project will be documented in this file.
|
All notable changes to this project will be documented in this file.
|
||||||
|
|
||||||
## [3.0.0] - 2025-11-22
|
## [Unreleased]
|
||||||
- 支持 .NET 10.0
|
|
||||||
- 增加 IPv6 支持
|
|
||||||
- 修复若干 bug
|
|
||||||
|
|
||||||
## [2.0.1] - 2023-07-30
|
## [2.0.1] - 2023-07-30
|
||||||
|
|
||||||
|
|
@ -21,4 +18,4 @@ All notable changes to this project will be documented in this file.
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
- Dependent file query policies CachePolicy.VectorIndex, CachePolicy.File support thread-safe concurrent queries
|
- Dependent file query policies CachePolicy.VectorIndex, CachePolicy.File support thread-safe concurrent queries
|
||||||
- Dramatically optimizes overall performance
|
- Dramatically optimizes overall performance
|
||||||
|
|
@ -1,62 +0,0 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
|
||||||
// @Date 2023/07/25
|
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
using BenchmarkDotNet.Attributes;
|
|
||||||
using IP2Region.Net.XDB;
|
|
||||||
|
|
||||||
namespace IP2Region.Net.BenchMark;
|
|
||||||
|
|
||||||
[MemoryDiagnoser]
|
|
||||||
public class Benchmarks
|
|
||||||
{
|
|
||||||
private static readonly string XdbPathV4 = Path.Combine(AppContext.BaseDirectory, "IP2Region", "ip2region_v4.xdb");
|
|
||||||
private static readonly string XdbPathV6 = Path.Combine(AppContext.BaseDirectory, "IP2Region", "ip2region_v6.xdb");
|
|
||||||
private static readonly Searcher _contentV4Searcher = new(CachePolicy.Content, XdbPathV4);
|
|
||||||
private static readonly Searcher _vectorV4Searcher = new(CachePolicy.VectorIndex, XdbPathV4);
|
|
||||||
private static readonly Searcher _fileV4Searcher = new(CachePolicy.File, XdbPathV4);
|
|
||||||
private static readonly Searcher _contentV6Searcher = new(CachePolicy.Content, XdbPathV6);
|
|
||||||
private static readonly Searcher _vectorV6Searcher = new(CachePolicy.VectorIndex, XdbPathV6);
|
|
||||||
private static readonly Searcher _fileV6Searcher = new(CachePolicy.File, XdbPathV6);
|
|
||||||
|
|
||||||
private readonly string _testIPv4Address = "114.114.114.114";
|
|
||||||
private readonly string _testIPv6Address = "240e:3b7:3272:d8d0:db09:c067:8d59:539e";
|
|
||||||
|
|
||||||
public Benchmarks()
|
|
||||||
{
|
|
||||||
_contentV4Searcher.Search(_testIPv4Address);
|
|
||||||
_vectorV4Searcher.Search(_testIPv4Address);
|
|
||||||
_fileV4Searcher.Search(_testIPv4Address);
|
|
||||||
|
|
||||||
_contentV6Searcher.Search(_testIPv6Address);
|
|
||||||
_vectorV6Searcher.Search(_testIPv6Address);
|
|
||||||
_fileV6Searcher.Search(_testIPv6Address);
|
|
||||||
}
|
|
||||||
|
|
||||||
[Benchmark]
|
|
||||||
[BenchmarkCategory("IPv4")]
|
|
||||||
public void ContentIPv4() => _contentV4Searcher.Search(_testIPv4Address);
|
|
||||||
|
|
||||||
[Benchmark]
|
|
||||||
[BenchmarkCategory("IPv4")]
|
|
||||||
public void VectorIPv4() => _vectorV4Searcher.Search(_testIPv4Address);
|
|
||||||
|
|
||||||
[Benchmark]
|
|
||||||
[BenchmarkCategory("IPv4")]
|
|
||||||
public void FileIPv4() => _fileV4Searcher.Search(_testIPv4Address);
|
|
||||||
|
|
||||||
[Benchmark]
|
|
||||||
[BenchmarkCategory("IPv6")]
|
|
||||||
public void ContentIPv6() => _contentV6Searcher.Search(_testIPv6Address);
|
|
||||||
|
|
||||||
[Benchmark]
|
|
||||||
[BenchmarkCategory("IPv6")]
|
|
||||||
public void VectorIPv6() => _vectorV6Searcher.Search(_testIPv6Address);
|
|
||||||
|
|
||||||
[Benchmark]
|
|
||||||
[BenchmarkCategory("IPv6")]
|
|
||||||
public void FileIPv6() => _fileV6Searcher.Search(_testIPv6Address);
|
|
||||||
}
|
|
||||||
|
|
@ -2,14 +2,14 @@
|
||||||
|
|
||||||
<PropertyGroup>
|
<PropertyGroup>
|
||||||
<OutputType>Exe</OutputType>
|
<OutputType>Exe</OutputType>
|
||||||
<TargetFramework>net10.0</TargetFramework>
|
<TargetFramework>net6.0</TargetFramework>
|
||||||
<ImplicitUsings>enable</ImplicitUsings>
|
<ImplicitUsings>enable</ImplicitUsings>
|
||||||
<Nullable>enable</Nullable>
|
<Nullable>enable</Nullable>
|
||||||
</PropertyGroup>
|
</PropertyGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<PackageReference Include="BenchmarkDotNet" Version="0.15.6" />
|
<PackageReference Include="BenchmarkDotNet" Version="0.13.2" />
|
||||||
<PackageReference Include="BenchmarkDotNet.Annotations" Version="0.15.6" />
|
<PackageReference Include="BenchmarkDotNet.Annotations" Version="0.13.2" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
|
|
@ -17,14 +17,10 @@
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<Content Include="..\..\..\data\ip2region_v4.xdb">
|
<Content Include="..\..\..\data\ip2region.xdb">
|
||||||
<Link>IP2Region/ip2region_v4.xdb</Link>
|
<Link>IP2Region/ip2region.xdb</Link>
|
||||||
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
|
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
|
||||||
</Content>
|
</Content>
|
||||||
<Content Include="..\..\..\data\ip2region_v6.xdb">
|
</ItemGroup>
|
||||||
<Link>IP2Region/ip2region_v6.xdb</Link>
|
|
||||||
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
|
|
||||||
</Content>
|
|
||||||
</ItemGroup>
|
|
||||||
|
|
||||||
</Project>
|
</Project>
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,29 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
using BenchmarkDotNet.Attributes;
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
|
||||||
// @Date 2023/07/25
|
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
using BenchmarkDotNet.Running;
|
using BenchmarkDotNet.Running;
|
||||||
using IP2Region.Net.BenchMark;
|
using IP2Region.Net.Abstractions;
|
||||||
|
using IP2Region.Net.XDB;
|
||||||
|
|
||||||
BenchmarkRunner.Run<Benchmarks>();
|
BenchmarkRunner.Run(typeof(Program).Assembly);
|
||||||
|
|
||||||
|
public class CachePolicyCompare
|
||||||
|
{
|
||||||
|
private static readonly string XdbPath = Path.Combine(AppContext.BaseDirectory, "IP2Region", "ip2region.xdb");
|
||||||
|
private readonly ISearcher _contentSearcher = new Searcher(CachePolicy.Content, XdbPath);
|
||||||
|
private readonly ISearcher _vectorSearcher = new Searcher(CachePolicy.VectorIndex,XdbPath);
|
||||||
|
private readonly ISearcher _fileSearcher = new Searcher(CachePolicy.File,XdbPath);
|
||||||
|
|
||||||
|
private readonly string _testIpAddress = "114.114.114.114";
|
||||||
|
|
||||||
|
[Benchmark]
|
||||||
|
[BenchmarkCategory(nameof(CachePolicy.Content))]
|
||||||
|
public void CachePolicy_Content() => _contentSearcher.Search(_testIpAddress);
|
||||||
|
|
||||||
|
[Benchmark]
|
||||||
|
[BenchmarkCategory(nameof(CachePolicy.VectorIndex))]
|
||||||
|
public void CachePolicy_VectorIndex() => _vectorSearcher.Search(_testIpAddress);
|
||||||
|
|
||||||
|
|
||||||
|
[Benchmark]
|
||||||
|
[BenchmarkCategory(nameof(CachePolicy.File))]
|
||||||
|
public void CachePolicy_File() => _fileSearcher.Search(_testIpAddress);
|
||||||
|
}
|
||||||
|
|
@ -1,51 +1,34 @@
|
||||||
<Project Sdk="Microsoft.NET.Sdk">
|
<Project Sdk="Microsoft.NET.Sdk">
|
||||||
|
|
||||||
<PropertyGroup>
|
<PropertyGroup>
|
||||||
<TargetFramework>net10.0</TargetFramework>
|
<TargetFramework>net6.0</TargetFramework>
|
||||||
<ImplicitUsings>enable</ImplicitUsings>
|
<ImplicitUsings>enable</ImplicitUsings>
|
||||||
<Nullable>enable</Nullable>
|
<Nullable>enable</Nullable>
|
||||||
|
|
||||||
<IsPackable>false</IsPackable>
|
<IsPackable>false</IsPackable>
|
||||||
</PropertyGroup>
|
</PropertyGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="18.0.1" />
|
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.1.0" />
|
||||||
<PackageReference Include="xunit" Version="2.*" />
|
<PackageReference Include="NUnit" Version="3.13.3" />
|
||||||
<PackageReference Include="xunit.runner.visualstudio" Version="3.*">
|
<PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
|
||||||
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
<PackageReference Include="NUnit.Analyzers" Version="3.3.0" />
|
||||||
<PrivateAssets>all</PrivateAssets>
|
<PackageReference Include="coverlet.collector" Version="3.1.2" />
|
||||||
</PackageReference>
|
</ItemGroup>
|
||||||
<PackageReference Include="coverlet.collector" Version="6.0.4">
|
|
||||||
<PrivateAssets>all</PrivateAssets>
|
|
||||||
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
|
||||||
</PackageReference>
|
|
||||||
</ItemGroup>
|
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="10.0.0" />
|
<ProjectReference Include="..\IP2Region.Net\IP2Region.Net.csproj" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<ProjectReference Include="..\IP2Region.Net\IP2Region.Net.csproj" />
|
<Content Include="..\..\..\data\ip.merge.txt">
|
||||||
</ItemGroup>
|
<Link>TestData/ip.merge.txt</Link>
|
||||||
|
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
|
||||||
<ItemGroup>
|
</Content>
|
||||||
<Content Include="..\..\..\data\ipv4_source.txt">
|
<Content Include="..\..\..\data\ip2region.xdb">
|
||||||
<Link>TestData/ipv4_source.txt</Link>
|
<Link>TestData/ip2region.xdb</Link>
|
||||||
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
|
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
|
||||||
</Content>
|
</Content>
|
||||||
<Content Include="..\..\..\data\ip2region_v4.xdb">
|
</ItemGroup>
|
||||||
<Link>TestData/ip2region_v4.xdb</Link>
|
|
||||||
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
|
|
||||||
</Content>
|
|
||||||
<Content Include="..\..\..\data\ipv6_source.txt">
|
|
||||||
<Link>TestData/ipv6_source.txt</Link>
|
|
||||||
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
|
|
||||||
</Content>
|
|
||||||
<Content Include="..\..\..\data\ip2region_v6.xdb">
|
|
||||||
<Link>TestData/ip2region_v6.xdb</Link>
|
|
||||||
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
|
|
||||||
</Content>
|
|
||||||
</ItemGroup>
|
|
||||||
|
|
||||||
</Project>
|
</Project>
|
||||||
|
|
|
||||||
|
|
@ -1,163 +1,80 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
|
||||||
// @Date 2023/07/25
|
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
using IP2Region.Net.Abstractions;
|
|
||||||
using IP2Region.Net.XDB;
|
using IP2Region.Net.XDB;
|
||||||
using Microsoft.Extensions.DependencyInjection;
|
|
||||||
using System.Net;
|
|
||||||
using Xunit;
|
|
||||||
|
|
||||||
namespace IP2Region.Net.Test;
|
namespace IP2Region.Net.Test;
|
||||||
|
|
||||||
|
[TestFixture]
|
||||||
public class SearcherTest
|
public class SearcherTest
|
||||||
{
|
{
|
||||||
private readonly string _xdbPathV4 = Path.Combine(AppContext.BaseDirectory, "TestData", "ip2region_v4.xdb");
|
private readonly string _xdbPath = Path.Combine(AppContext.BaseDirectory, "TestData", "ip2region.xdb");
|
||||||
private readonly string _xdbPathV6 = Path.Combine(AppContext.BaseDirectory, "TestData", "ip2region_v6.xdb");
|
|
||||||
|
public static IEnumerable<string> Ips()
|
||||||
[Theory]
|
|
||||||
[InlineData("58.251.27.201", "中国|广东省|深圳市|联通|CN", "v4")]
|
|
||||||
[InlineData("114.114.114.114", "中国|江苏省|南京市|0|CN", "v4")]
|
|
||||||
[InlineData("119.29.29.29", "中国|北京|北京市|腾讯|CN", "v4")]
|
|
||||||
[InlineData("223.5.5.5", "中国|浙江省|杭州市|阿里|CN", "v4")]
|
|
||||||
[InlineData("180.76.76.76", "中国|北京|北京市|百度|CN", "v4")]
|
|
||||||
[InlineData("8.8.8.8", "United States|California|0|Google LLC|US", "v4")]
|
|
||||||
[InlineData("240e:3b7:3272:d8d0:db09:c067:8d59:539e", "中国|广东省|深圳市|电信|CN", "v6")]
|
|
||||||
public void TestSearchCacheContent(string ip, string expected, string version)
|
|
||||||
{
|
{
|
||||||
var _xdbPath = version == "v4" ? _xdbPathV4 : _xdbPathV6;
|
yield return "114.114.114.114";
|
||||||
var contentSearcher = new Searcher(CachePolicy.Content, _xdbPath);
|
yield return "119.29.29.29";
|
||||||
|
yield return "223.5.5.5";
|
||||||
|
yield return "180.76.76.76";
|
||||||
|
yield return "8.8.8.8";
|
||||||
|
}
|
||||||
|
|
||||||
|
[TestCaseSource(nameof(Ips))]
|
||||||
|
[Parallelizable(ParallelScope.All)]
|
||||||
|
public void TestSearchCacheContent(string ip)
|
||||||
|
{
|
||||||
|
var contentSearcher = new Searcher(CachePolicy.Content,_xdbPath);
|
||||||
var region = contentSearcher.Search(ip);
|
var region = contentSearcher.Search(ip);
|
||||||
Assert.Equal(expected, region);
|
Console.WriteLine(region);
|
||||||
}
|
}
|
||||||
|
|
||||||
[Theory]
|
[TestCaseSource(nameof(Ips))]
|
||||||
[InlineData("58.251.27.201", "中国|广东省|深圳市|联通|CN", "v4")]
|
[Parallelizable(ParallelScope.All)]
|
||||||
[InlineData("114.114.114.114", "中国|江苏省|南京市|0|CN", "v4")]
|
public void TestSearchCacheVector(string ip)
|
||||||
[InlineData("119.29.29.29", "中国|北京|北京市|腾讯|CN", "v4")]
|
|
||||||
[InlineData("223.5.5.5", "中国|浙江省|杭州市|阿里|CN", "v4")]
|
|
||||||
[InlineData("180.76.76.76", "中国|北京|北京市|百度|CN", "v4")]
|
|
||||||
[InlineData("8.8.8.8", "United States|California|0|Google LLC|US", "v4")]
|
|
||||||
[InlineData("240e:3b7:3272:d8d0:db09:c067:8d59:539e", "中国|广东省|深圳市|电信|CN", "v6")]
|
|
||||||
public void TestSearchCacheVector(string ip, string expected, string version)
|
|
||||||
{
|
{
|
||||||
var _xdbPath = version == "v4" ? _xdbPathV4 : _xdbPathV6;
|
var vectorSearcher = new Searcher(CachePolicy.VectorIndex,_xdbPath);
|
||||||
var vectorSearcher = new Searcher(CachePolicy.VectorIndex, _xdbPath);
|
|
||||||
var region = vectorSearcher.Search(ip);
|
var region = vectorSearcher.Search(ip);
|
||||||
Assert.Equal(expected, region);
|
Console.WriteLine(region);
|
||||||
}
|
}
|
||||||
|
|
||||||
[Theory]
|
[TestCaseSource(nameof(Ips))]
|
||||||
[InlineData("58.251.0.0", "中国|广东省|深圳市|联通|CN", "v4")]
|
[Parallelizable(ParallelScope.All)]
|
||||||
[InlineData("58.251.255.255", "中国|广东省|深圳市|联通|CN", "v4")]
|
public void TestSearchCacheFile(string ip)
|
||||||
[InlineData("58.251.27.201", "中国|广东省|深圳市|联通|CN", "v4")]
|
|
||||||
[InlineData("114.114.114.114", "中国|江苏省|南京市|0|CN", "v4")]
|
|
||||||
[InlineData("119.29.29.29", "中国|北京|北京市|腾讯|CN", "v4")]
|
|
||||||
[InlineData("223.5.5.5", "中国|浙江省|杭州市|阿里|CN", "v4")]
|
|
||||||
[InlineData("180.76.76.76", "中国|北京|北京市|百度|CN", "v4")]
|
|
||||||
[InlineData("8.8.8.8", "United States|California|0|Google LLC|US", "v4")]
|
|
||||||
[InlineData("240e:3b7:3272:d8d0:db09:c067:8d59:539e", "中国|广东省|深圳市|电信|CN", "v6")]
|
|
||||||
[InlineData("240e:044d:2d00:0000:0000:0000:0000:0000", "中国|云南|楚雄|电信|CN", "v6")]
|
|
||||||
public void TestSearchCacheFile(string ip, string expected, string version)
|
|
||||||
{
|
{
|
||||||
var _xdbPath = version == "v4" ? _xdbPathV4 : _xdbPathV6;
|
var fileSearcher = new Searcher(CachePolicy.File,_xdbPath);
|
||||||
var fileSearcher = new Searcher(CachePolicy.File, _xdbPath);
|
|
||||||
var region = fileSearcher.Search(ip);
|
var region = fileSearcher.Search(ip);
|
||||||
Assert.Equal(expected, region);
|
Console.WriteLine(region);
|
||||||
}
|
}
|
||||||
|
|
||||||
[Fact]
|
[TestCase(CachePolicy.Content)]
|
||||||
public void IoCount_File_Ok()
|
[TestCase(CachePolicy.VectorIndex)]
|
||||||
|
[TestCase(CachePolicy.File)]
|
||||||
|
public void TestBenchSearch(CachePolicy cachePolicy)
|
||||||
{
|
{
|
||||||
var searcher = new Searcher(CachePolicy.File, _xdbPathV4);
|
Searcher searcher = new Searcher(cachePolicy,_xdbPath);
|
||||||
searcher.Search("58.251.27.201");
|
var srcPath = Path.Combine(AppContext.BaseDirectory, "TestData", "ip.merge.txt");
|
||||||
Assert.Equal(3, searcher.IoCount);
|
|
||||||
|
|
||||||
searcher.Search("58.251.27.201");
|
|
||||||
Assert.Equal(3, searcher.IoCount);
|
|
||||||
|
|
||||||
searcher.Dispose();
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void IoCount_Vector_Ok()
|
|
||||||
{
|
|
||||||
var searcher = new Searcher(CachePolicy.VectorIndex, _xdbPathV4);
|
|
||||||
searcher.Search("58.251.27.201");
|
|
||||||
Assert.Equal(2, searcher.IoCount);
|
|
||||||
|
|
||||||
searcher.Search("58.251.27.201");
|
|
||||||
Assert.Equal(2, searcher.IoCount);
|
|
||||||
|
|
||||||
searcher.Dispose();
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void IoCount_Content_Ok()
|
|
||||||
{
|
|
||||||
var searcher = new Searcher(CachePolicy.Content, _xdbPathV4);
|
|
||||||
searcher.Search("58.251.27.201");
|
|
||||||
Assert.Equal(0, searcher.IoCount);
|
|
||||||
|
|
||||||
searcher.Search("58.251.27.201");
|
|
||||||
Assert.Equal(0, searcher.IoCount);
|
|
||||||
|
|
||||||
searcher.Dispose();
|
|
||||||
}
|
|
||||||
|
|
||||||
[Theory]
|
|
||||||
[InlineData("58.251.255.255", "中国|广东省|深圳市|联通|CN")]
|
|
||||||
public void Search_Ip_Ok(string ipStr, string expected)
|
|
||||||
{
|
|
||||||
var fileSearcher = new Searcher(CachePolicy.File, _xdbPathV4);
|
|
||||||
var ipAddress = IPAddress.Parse(ipStr);
|
|
||||||
var region = fileSearcher.Search(ipAddress);
|
|
||||||
Assert.Equal(expected, region);
|
|
||||||
}
|
|
||||||
|
|
||||||
[Theory]
|
|
||||||
[InlineData("58.251.255.255", "中国|广东省|深圳市|联通|CN")]
|
|
||||||
public void AddIP2RegionService_Ok(string ipStr, string expected)
|
|
||||||
{
|
|
||||||
var services = new ServiceCollection();
|
|
||||||
services.AddIP2RegionService(_xdbPathV4, CachePolicy.File);
|
|
||||||
|
|
||||||
var provider = services.BuildServiceProvider();
|
|
||||||
var searcher = provider.GetRequiredService<ISearcher>();
|
|
||||||
var region = searcher.Search(ipStr);
|
|
||||||
Assert.Equal(expected, region);
|
|
||||||
|
|
||||||
searcher = provider.GetRequiredKeyedService<ISearcher>("IP2Region.Net");
|
|
||||||
region = searcher.Search(ipStr);
|
|
||||||
Assert.Equal(expected, region);
|
|
||||||
}
|
|
||||||
|
|
||||||
[Theory]
|
|
||||||
[InlineData(CachePolicy.Content, "v4")]
|
|
||||||
[InlineData(CachePolicy.VectorIndex, "v4")]
|
|
||||||
[InlineData(CachePolicy.File, "v4")]
|
|
||||||
[InlineData(CachePolicy.Content, "v6")]
|
|
||||||
[InlineData(CachePolicy.VectorIndex, "v6")]
|
|
||||||
[InlineData(CachePolicy.File, "v6")]
|
|
||||||
public void TestBenchSearch(CachePolicy cachePolicy, string version)
|
|
||||||
{
|
|
||||||
var _xdbPath = version == "v4" ? _xdbPathV4 : _xdbPathV6;
|
|
||||||
var searcher = new Searcher(cachePolicy, _xdbPath);
|
|
||||||
var srcPath = Path.Combine(AppContext.BaseDirectory, "TestData", $"ip{version}_source.txt");
|
|
||||||
|
|
||||||
foreach (var line in File.ReadLines(srcPath))
|
foreach (var line in File.ReadLines(srcPath))
|
||||||
{
|
{
|
||||||
var ps = line.Trim().Split("|", 3);
|
var ps = line.Trim().Split("|", 3);
|
||||||
var sip = ps[0];
|
|
||||||
var eip = ps[1];
|
|
||||||
|
|
||||||
var s1 = searcher.Search(sip);
|
if (ps.Length != 3)
|
||||||
var s2 = searcher.Search(eip);
|
{
|
||||||
Assert.Equal(s1, ps[2]);
|
throw new ArgumentException($"invalid ip segment line {line}", nameof(line));
|
||||||
Assert.Equal(s2, ps[2]);
|
}
|
||||||
|
|
||||||
|
var sip = Util.IpAddressToUInt32(ps[0]);
|
||||||
|
var eip = Util.IpAddressToUInt32(ps[1]);
|
||||||
|
var mip = Util.GetMidIp(sip, eip);
|
||||||
|
|
||||||
|
uint[] temp = { sip, Util.GetMidIp(sip, mip), mip, Util.GetMidIp(mip, eip), eip };
|
||||||
|
|
||||||
|
foreach (var ip in temp)
|
||||||
|
{
|
||||||
|
var region = searcher.Search(ip);
|
||||||
|
|
||||||
|
if (region != ps[2])
|
||||||
|
{
|
||||||
|
throw new Exception($"failed search {ip} with ({region}!={ps[2]})");
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -0,0 +1 @@
|
||||||
|
global using NUnit.Framework;
|
||||||
|
|
@ -1,27 +1,14 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
using IP2Region.Net.XDB;
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
|
||||||
// @Date 2023/07/25
|
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
using Xunit;
|
|
||||||
|
|
||||||
namespace IP2Region.Net.Test;
|
namespace IP2Region.Net.Test;
|
||||||
|
|
||||||
|
[TestFixture]
|
||||||
public class UtilTest
|
public class UtilTest
|
||||||
{
|
{
|
||||||
[Fact]
|
[TestCase("114.114.114.114")]
|
||||||
public void IpAddressToUInt32_Ok()
|
public void TestIpAddressToUInt32(string value)
|
||||||
{
|
{
|
||||||
var uintIp = XDB.Util.IpAddressToUInt32("114.114.114.114");
|
var uintIp = XDB.Util.IpAddressToUInt32(value);
|
||||||
Assert.Equal((uint)1920103026, uintIp);
|
Console.WriteLine(uintIp);
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public void GetMidIp_Ok()
|
|
||||||
{
|
|
||||||
var uintIp = XDB.Util.GetMidIp(1, 10);
|
|
||||||
Assert.Equal((uint)5, uintIp);
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -1,49 +0,0 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
|
||||||
// @Date 2023/07/25
|
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
using Xunit;
|
|
||||||
|
|
||||||
namespace IP2Region.Net.Test;
|
|
||||||
|
|
||||||
public class XdbTest
|
|
||||||
{
|
|
||||||
[Fact]
|
|
||||||
public async Task VersionIPV4_Ok()
|
|
||||||
{
|
|
||||||
var db = Path.Combine(AppContext.BaseDirectory, "TestData", $"ip2region_v4.xdb");
|
|
||||||
|
|
||||||
var version = await XDB.Util.GetVersionAsync(db);
|
|
||||||
Assert.Equal(3, version.Ver);
|
|
||||||
Assert.Equal(1, version.CachePolice);
|
|
||||||
//Assert.Equal("2025-09-06 02:24:16", version.CreatedTime.ToString("yyyy-MM-dd HH:mm:ss"));
|
|
||||||
//Assert.Equal((uint)955933, version.StartIndex);
|
|
||||||
//Assert.Equal((uint)11042415, version.EndIndex);
|
|
||||||
Assert.Equal(4, version.IPVer);
|
|
||||||
Assert.Equal(4, version.BytesCount);
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public async Task VersionIPV6_Ok()
|
|
||||||
{
|
|
||||||
var db = Path.Combine(AppContext.BaseDirectory, "TestData", $"ip2region_v6.xdb");
|
|
||||||
var version = await XDB.Util.GetVersionAsync(db);
|
|
||||||
Assert.Equal(3, version.Ver);
|
|
||||||
Assert.Equal(1, version.CachePolice);
|
|
||||||
//Assert.Equal("2025-10-17 04:41:04", version.CreatedTime.ToString("yyyy-MM-dd HH:mm:ss"));
|
|
||||||
//Assert.Equal((uint)3094259, version.StartIndex);
|
|
||||||
//Assert.Equal((uint)36258303, version.EndIndex);
|
|
||||||
Assert.Equal(6, version.IPVer);
|
|
||||||
Assert.Equal(4, version.BytesCount);
|
|
||||||
}
|
|
||||||
|
|
||||||
[Fact]
|
|
||||||
public async Task GetVersionAsync_Error()
|
|
||||||
{
|
|
||||||
await Assert.ThrowsAsync<ArgumentNullException>(async () => await XDB.Util.GetVersionAsync(null!));
|
|
||||||
await Assert.ThrowsAsync<FileNotFoundException>(async () => await XDB.Util.GetVersionAsync(Path.Combine(AppContext.BaseDirectory, "test.xdb")));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,9 +0,0 @@
|
||||||
<Solution>
|
|
||||||
<Folder Name="/Solution Items/">
|
|
||||||
<File Path=".editorconfig" />
|
|
||||||
<File Path="README.md" />
|
|
||||||
</Folder>
|
|
||||||
<Project Path="IP2Region.Net.BenchMark/IP2Region.Net.BenchMark.csproj" />
|
|
||||||
<Project Path="IP2Region.Net.Test/IP2Region.Net.Test.csproj" />
|
|
||||||
<Project Path="IP2Region.Net/IP2Region.Net.csproj" />
|
|
||||||
</Solution>
|
|
||||||
|
|
@ -1,40 +1,20 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
// Copyright 2023 The Ip2Region Authors. All rights reserved.
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
// Use of this source code is governed by a Apache2.0-style
|
||||||
// license that can be found in the LICENSE file.
|
// license that can be found in the LICENSE file.
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
// @Author Alan <lzh.shap@gmail.com>
|
||||||
// @Date 2023/07/25
|
// @Date 2023/07/25
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
using System.Net;
|
using System.Net;
|
||||||
|
|
||||||
namespace IP2Region.Net.Abstractions;
|
namespace IP2Region.Net.Abstractions;
|
||||||
|
|
||||||
/// <summary>
|
public interface ISearcher
|
||||||
/// IP 转化为地理位置搜索器接口
|
|
||||||
/// </summary>
|
|
||||||
public interface ISearcher : IDisposable
|
|
||||||
{
|
{
|
||||||
/// <summary>
|
|
||||||
/// 搜索方法
|
|
||||||
/// </summary>
|
|
||||||
/// <param name="ipStr">IP 地址字符串 如 192.168.0.1</param>
|
|
||||||
/// <returns></returns>
|
|
||||||
string? Search(string ipStr);
|
string? Search(string ipStr);
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 搜索方法
|
|
||||||
/// </summary>
|
|
||||||
string? Search(IPAddress ipAddress);
|
string? Search(IPAddress ipAddress);
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 搜索方法 仅限 IPv4 使用
|
|
||||||
/// </summary>
|
|
||||||
/// <param name="ipAddress">IPv4 地址字节数组小端读取 uint 数值</param>
|
|
||||||
[Obsolete("已弃用，请改用其他方法；Deprecated; please use Search(string) or Search(IPAddress) method.")]
|
|
||||||
string? Search(uint ipAddress);
|
string? Search(uint ipAddress);
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 获得 内部 IO 访问次数
|
|
||||||
/// </summary>
|
|
||||||
int IoCount { get; }
|
int IoCount { get; }
|
||||||
}
|
}
|
||||||
|
|
@ -1,40 +0,0 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
|
||||||
// @Date 2023/07/25
|
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
using IP2Region.Net.Abstractions;
|
|
||||||
using IP2Region.Net.XDB;
|
|
||||||
using Microsoft.Extensions.DependencyInjection.Extensions;
|
|
||||||
|
|
||||||
namespace Microsoft.Extensions.DependencyInjection;
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// IP2Region 服务扩展类
|
|
||||||
/// </summary>
|
|
||||||
public static class IP2RegionExtensions
|
|
||||||
{
|
|
||||||
/// <summary>
|
|
||||||
/// 添加 IP2RegionService 服务。
|
|
||||||
/// </summary>
|
|
||||||
/// <param name="services"><see cref="IServiceCollection"/> 集合</param>
|
|
||||||
/// <param name="path">IP2Region 数据库文件的路径。</param>
|
|
||||||
/// <param name="cachePolicy">缓存策略，默认为 <see cref="CachePolicy.Content"/>。</param>
|
|
||||||
public static IServiceCollection AddIP2RegionService(this IServiceCollection services, string path, CachePolicy cachePolicy = CachePolicy.Content)
|
|
||||||
{
|
|
||||||
services.TryAddSingleton<ISearcher>(provider =>
|
|
||||||
{
|
|
||||||
return new Searcher(cachePolicy, path);
|
|
||||||
});
|
|
||||||
#if NET8_0_OR_GREATER
|
|
||||||
services.TryAddKeyedSingleton("IP2Region.Net", (provider, _) =>
|
|
||||||
{
|
|
||||||
return provider.GetRequiredService<ISearcher>();
|
|
||||||
});
|
|
||||||
#endif
|
|
||||||
|
|
||||||
return services;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -2,9 +2,9 @@
|
||||||
|
|
||||||
<PropertyGroup>
|
<PropertyGroup>
|
||||||
<id>IP2Region.Net</id>
|
<id>IP2Region.Net</id>
|
||||||
<version>3.0.2</version>
|
<version>2.0.2</version>
|
||||||
<title>IP2Region.Net</title>
|
<title>IP2Region.Net</title>
|
||||||
<authors>Alan Lee;Argo Zhang(argo@live.ca)</authors>
|
<authors>Alan Lee</authors>
|
||||||
<PackageLicenseExpression>Apache-2.0</PackageLicenseExpression>
|
<PackageLicenseExpression>Apache-2.0</PackageLicenseExpression>
|
||||||
<PackageReadmeFile>README.md</PackageReadmeFile>
|
<PackageReadmeFile>README.md</PackageReadmeFile>
|
||||||
<PackageProjectUrl>https://github.com/lionsoul2014/ip2region/tree/master/binding/csharp</PackageProjectUrl>
|
<PackageProjectUrl>https://github.com/lionsoul2014/ip2region/tree/master/binding/csharp</PackageProjectUrl>
|
||||||
|
|
@ -15,49 +15,18 @@
|
||||||
<RepositoryType>git</RepositoryType>
|
<RepositoryType>git</RepositoryType>
|
||||||
<ImplicitUsings>enable</ImplicitUsings>
|
<ImplicitUsings>enable</ImplicitUsings>
|
||||||
<Nullable>enable</Nullable>
|
<Nullable>enable</Nullable>
|
||||||
<TargetFrameworks>netstandard2.0;netstandard2.1;net6.0;net7.0;net8.0;net9.0;net10.0</TargetFrameworks>
|
<TargetFrameworks>netstandard2.0;netstandard2.1;net6.0;net7.0</TargetFrameworks>
|
||||||
<LangVersion>latest</LangVersion>
|
<LangVersion>10.0</LangVersion>
|
||||||
<UserSecretsId>c2f07fe1-a300-4de3-8200-1278ed8cb5b7</UserSecretsId>
|
<UserSecretsId>c2f07fe1-a300-4de3-8200-1278ed8cb5b7</UserSecretsId>
|
||||||
</PropertyGroup>
|
</PropertyGroup>
|
||||||
|
|
||||||
<ItemGroup>
|
<ItemGroup>
|
||||||
<None Include="..\README.md" Pack="true" PackagePath="\" />
|
<None Include="..\README.md" Pack="true" PackagePath="\" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
|
||||||
<ItemGroup Condition="'$(TargetFramework)' == 'netstandard2.0'">
|
<ItemGroup Condition="'$(TargetFramework)' == 'netstandard2.0'">
|
||||||
<PackageReference Include="System.Buffers" Version="4.5.1" />
|
<PackageReference Include="System.Buffers" Version="4.5.1" />
|
||||||
<PackageReference Include="System.Memory" Version="4.5.5" />
|
<PackageReference Include="System.Memory" Version="4.5.5" />
|
||||||
</ItemGroup>
|
</ItemGroup>
|
||||||
|
<ItemGroup>
|
||||||
<ItemGroup Condition="'$(TargetFramework)' == 'netstandard2.0'">
|
|
||||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.*" />
|
|
||||||
</ItemGroup>
|
|
||||||
|
|
||||||
<ItemGroup Condition="'$(TargetFramework)' == 'netstandard2.1'">
|
|
||||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.0" />
|
|
||||||
</ItemGroup>
|
|
||||||
|
|
||||||
<ItemGroup Condition="'$(TargetFramework)' == 'net6.0'">
|
|
||||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="8.0.*" />
|
|
||||||
</ItemGroup>
|
|
||||||
|
|
||||||
<ItemGroup Condition="'$(TargetFramework)' == 'net7.0'">
|
|
||||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="8.0.*" />
|
|
||||||
</ItemGroup>
|
|
||||||
|
|
||||||
<ItemGroup Condition="'$(TargetFramework)' == 'net8.0'">
|
|
||||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.*" />
|
|
||||||
</ItemGroup>
|
|
||||||
|
|
||||||
<ItemGroup Condition="'$(TargetFramework)' == 'net9.0'">
|
|
||||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.*" />
|
|
||||||
</ItemGroup>
|
|
||||||
|
|
||||||
<ItemGroup Condition="'$(TargetFramework)' == 'net10.0'">
|
|
||||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="10.0.*" />
|
|
||||||
</ItemGroup>
|
|
||||||
|
|
||||||
<ItemGroup>
|
|
||||||
<None Include="..\CHANGELOG.md">
|
<None Include="..\CHANGELOG.md">
|
||||||
<Link>CHANGELOG.md</Link>
|
<Link>CHANGELOG.md</Link>
|
||||||
</None>
|
</None>
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,64 @@
|
||||||
|
// Copyright 2023 The Ip2Region Authors. All rights reserved.
|
||||||
|
// Use of this source code is governed by a Apache2.0-style
|
||||||
|
// license that can be found in the LICENSE file.
|
||||||
|
// @Author Alan <lzh.shap@gmail.com>
|
||||||
|
// @Date 2023/07/25
|
||||||
|
|
||||||
|
using System.Buffers;
|
||||||
|
|
||||||
|
namespace IP2Region.Net.Internal.Abstractions;
|
||||||
|
|
||||||
|
internal abstract class AbstractCacheStrategy
|
||||||
|
{
|
||||||
|
protected const int HeaderInfoLength = 256;
|
||||||
|
protected const int VectorIndexRows = 256;
|
||||||
|
protected const int VectorIndexCols = 256;
|
||||||
|
protected const int VectorIndexSize = 8;
|
||||||
|
|
||||||
|
protected readonly FileStream XdbFileStream;
|
||||||
|
private const int BufferSize = 4096;
|
||||||
|
|
||||||
|
internal int IoCount { get; private set; }
|
||||||
|
|
||||||
|
protected AbstractCacheStrategy(string xdbPath)
|
||||||
|
{
|
||||||
|
XdbFileStream = new FileStream(xdbPath, FileMode.Open, FileAccess.Read, FileShare.Read, BufferSize,
|
||||||
|
useAsync: true);
|
||||||
|
}
|
||||||
|
|
||||||
|
protected int GetVectorIndexStartPos(uint ip)
|
||||||
|
{
|
||||||
|
var il0 = ip >> 24 & 0xFF;
|
||||||
|
var il1 = ip >> 16 & 0xFF;
|
||||||
|
var idx = il0 * VectorIndexCols * VectorIndexSize + il1 * VectorIndexSize;
|
||||||
|
return (int)idx;
|
||||||
|
}
|
||||||
|
|
||||||
|
internal abstract ReadOnlyMemory<byte> GetVectorIndex(uint ip);
|
||||||
|
|
||||||
|
internal virtual ReadOnlyMemory<byte> GetData(int offset, int length)
|
||||||
|
{
|
||||||
|
byte[] buffer = ArrayPool<byte>.Shared.Rent(length);
|
||||||
|
int totalBytesRead = 0;
|
||||||
|
try
|
||||||
|
{
|
||||||
|
XdbFileStream.Seek(offset, SeekOrigin.Begin);
|
||||||
|
|
||||||
|
int bytesRead;
|
||||||
|
do
|
||||||
|
{
|
||||||
|
int bytesToRead = Math.Min(BufferSize, length - totalBytesRead);
|
||||||
|
bytesRead = XdbFileStream.Read(buffer, totalBytesRead, bytesToRead);
|
||||||
|
totalBytesRead += bytesRead;
|
||||||
|
|
||||||
|
IoCount++;
|
||||||
|
} while (bytesRead > 0 && totalBytesRead < length);
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
ArrayPool<byte>.Shared.Return(buffer);
|
||||||
|
}
|
||||||
|
|
||||||
|
return new ReadOnlyMemory<byte>(buffer, 0, totalBytesRead);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -1,20 +1,31 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
// Copyright 2023 The Ip2Region Authors. All rights reserved.
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
// Use of this source code is governed by a Apache2.0-style
|
||||||
// license that can be found in the LICENSE file.
|
// license that can be found in the LICENSE file.
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
// @Author Alan <lzh.shap@gmail.com>
|
||||||
// @Date 2023/07/25
|
// @Date 2023/07/25
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
|
using IP2Region.Net.Internal.Abstractions;
|
||||||
using IP2Region.Net.XDB;
|
using IP2Region.Net.XDB;
|
||||||
|
|
||||||
namespace IP2Region.Net.Internal;
|
namespace IP2Region.Net.Internal;
|
||||||
|
|
||||||
static class CacheStrategyFactory
|
internal class CacheStrategyFactory
|
||||||
{
|
{
|
||||||
public static ICacheStrategy CreateCacheStrategy(CachePolicy cachePolicy, string xdbPath) => cachePolicy switch
|
private readonly string _xdbPath;
|
||||||
|
|
||||||
|
public CacheStrategyFactory(string xdbPath)
|
||||||
{
|
{
|
||||||
CachePolicy.Content => new ContentCacheStrategy(xdbPath),
|
_xdbPath = xdbPath;
|
||||||
CachePolicy.VectorIndex => new VectorIndexCacheStrategy(xdbPath),
|
}
|
||||||
_ => new FileCacheStrategy(xdbPath),
|
|
||||||
};
|
public AbstractCacheStrategy CreateCacheStrategy(CachePolicy cachePolicy)
|
||||||
}
|
{
|
||||||
|
return cachePolicy switch
|
||||||
|
{
|
||||||
|
CachePolicy.Content => new ContentCacheStrategy(_xdbPath),
|
||||||
|
CachePolicy.VectorIndex => new VectorIndexCacheStrategy(_xdbPath),
|
||||||
|
CachePolicy.File => new FileCacheStrategy(_xdbPath),
|
||||||
|
_ => throw new ArgumentException(nameof(cachePolicy))
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -1,34 +1,32 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
// Copyright 2023 The Ip2Region Authors. All rights reserved.
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
// Use of this source code is governed by a Apache2.0-style
|
||||||
// license that can be found in the LICENSE file.
|
// license that can be found in the LICENSE file.
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
// @Author Alan <lzh.shap@gmail.com>
|
||||||
// @Date 2023/07/25
|
// @Date 2023/07/25
|
||||||
// Updated by Wong <vcd.hai@outlook.com> at 2025/12/31
|
|
||||||
|
using IP2Region.Net.Internal.Abstractions;
|
||||||
|
|
||||||
namespace IP2Region.Net.Internal;
|
namespace IP2Region.Net.Internal;
|
||||||
|
|
||||||
class ContentCacheStrategy(string xdbPath) : ICacheStrategy
|
internal class ContentCacheStrategy : AbstractCacheStrategy
|
||||||
{
|
{
|
||||||
// TODO: these constants can be moved to the interface as defaults when using .NET 10
|
private readonly ReadOnlyMemory<byte> _cacheData;
|
||||||
private const int HeaderInfoLength = 256;
|
|
||||||
private const int VectorIndexSize = 8;
|
|
||||||
|
|
||||||
private readonly ReadOnlyMemory<byte> _cacheData = File.ReadAllBytes(xdbPath);
|
public ContentCacheStrategy(string xdbPath) : base(xdbPath)
|
||||||
|
|
||||||
public int IoCount => 0;
|
|
||||||
|
|
||||||
public void ResetIoCount()
|
|
||||||
{
|
{
|
||||||
// Do nothing
|
_cacheData = base.GetData(0, (int)XdbFileStream.Length);
|
||||||
|
XdbFileStream.Close();
|
||||||
|
XdbFileStream.Dispose();
|
||||||
}
|
}
|
||||||
|
|
||||||
public ReadOnlyMemory<byte> GetVectorIndex(int offset)
|
internal override ReadOnlyMemory<byte> GetVectorIndex(uint ip)
|
||||||
=> _cacheData.Slice(HeaderInfoLength + offset, VectorIndexSize);
|
|
||||||
|
|
||||||
public ReadOnlyMemory<byte> GetData(long offset, int length) => _cacheData.Slice((int)offset, length);
|
|
||||||
|
|
||||||
public void Dispose()
|
|
||||||
{
|
{
|
||||||
// Do nothing
|
int idx = GetVectorIndexStartPos(ip);
|
||||||
|
return _cacheData.Slice(HeaderInfoLength + idx, VectorIndexSize);
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
internal override ReadOnlyMemory<byte> GetData(int offset, int length)
|
||||||
|
{
|
||||||
|
return _cacheData.Slice(offset, length);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -1,85 +1,22 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
// Copyright 2023 The Ip2Region Authors. All rights reserved.
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
// Use of this source code is governed by a Apache2.0-style
|
||||||
// license that can be found in the LICENSE file.
|
// license that can be found in the LICENSE file.
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
// @Author Alan <lzh.shap@gmail.com>
|
||||||
// @Date 2023/07/25
|
// @Date 2023/07/25
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
using System.Buffers;
|
using IP2Region.Net.Internal.Abstractions;
|
||||||
|
|
||||||
namespace IP2Region.Net.Internal;
|
namespace IP2Region.Net.Internal;
|
||||||
|
|
||||||
class FileCacheStrategy(string xdbPath) : ICacheStrategy
|
internal class FileCacheStrategy : AbstractCacheStrategy
|
||||||
{
|
{
|
||||||
protected const int HeaderInfoLength = 256;
|
public FileCacheStrategy(string xdbPath) : base(xdbPath)
|
||||||
protected const int VectorIndexSize = 8;
|
|
||||||
|
|
||||||
protected const int BufferSize = 64 * 1024;
|
|
||||||
|
|
||||||
protected FileStream XdbFileStream = new(xdbPath, FileMode.Open, FileAccess.Read, FileShare.Read, BufferSize, FileOptions.RandomAccess);
|
|
||||||
|
|
||||||
public int IoCount { get; set; }
|
|
||||||
|
|
||||||
public void ResetIoCount()
|
|
||||||
{
|
{
|
||||||
IoCount = 0;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
public virtual ReadOnlyMemory<byte> GetVectorIndex(int offset) => GetData(HeaderInfoLength + offset, VectorIndexSize);
|
internal override ReadOnlyMemory<byte> GetVectorIndex(uint ip)
|
||||||
|
|
||||||
public virtual ReadOnlyMemory<byte> GetData(long offset, int length)
|
|
||||||
{
|
{
|
||||||
var buffer = ArrayPool<byte>.Shared.Rent(length);
|
var idx = GetVectorIndexStartPos(ip);
|
||||||
try
|
return GetData(HeaderInfoLength + idx, VectorIndexSize);
|
||||||
{
|
|
||||||
int totalBytesRead = 0;
|
|
||||||
XdbFileStream.Seek(offset, SeekOrigin.Begin);
|
|
||||||
|
|
||||||
int bytesRead;
|
|
||||||
while (totalBytesRead < length)
|
|
||||||
{
|
|
||||||
bytesRead = XdbFileStream.Read(buffer, totalBytesRead, length - totalBytesRead);
|
|
||||||
if (bytesRead == 0)
|
|
||||||
{
|
|
||||||
break;
|
|
||||||
}
|
|
||||||
|
|
||||||
totalBytesRead += bytesRead;
|
|
||||||
IoCount++;
|
|
||||||
}
|
|
||||||
|
|
||||||
var ret = new byte[totalBytesRead];
|
|
||||||
if (totalBytesRead > 0)
|
|
||||||
{
|
|
||||||
Array.Copy(buffer, 0, ret, 0, totalBytesRead);
|
|
||||||
}
|
|
||||||
return ret;
|
|
||||||
}
|
|
||||||
finally
|
|
||||||
{
|
|
||||||
ArrayPool<byte>.Shared.Return(buffer);
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
}
|
||||||
/// <summary>
|
|
||||||
/// 释放文件句柄
|
|
||||||
/// </summary>
|
|
||||||
/// <param name="disposing"></param>
|
|
||||||
protected virtual void Dispose(bool disposing)
|
|
||||||
{
|
|
||||||
if (disposing)
|
|
||||||
{
|
|
||||||
XdbFileStream.Close();
|
|
||||||
XdbFileStream.Dispose();
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// <inheritdoc/>
|
|
||||||
/// </summary>
|
|
||||||
public void Dispose()
|
|
||||||
{
|
|
||||||
Dispose(true);
|
|
||||||
GC.SuppressFinalize(this);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,19 +0,0 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
|
||||||
// @Date 2023/07/25
|
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
namespace IP2Region.Net.Internal;
|
|
||||||
|
|
||||||
internal interface ICacheStrategy : IDisposable
|
|
||||||
{
|
|
||||||
int IoCount { get; }
|
|
||||||
|
|
||||||
void ResetIoCount();
|
|
||||||
|
|
||||||
ReadOnlyMemory<byte> GetVectorIndex(int offset);
|
|
||||||
|
|
||||||
ReadOnlyMemory<byte> GetData(long offset, int length);
|
|
||||||
}
|
|
||||||
|
|
@ -1,23 +1,26 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
// Copyright 2023 The Ip2Region Authors. All rights reserved.
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
// Use of this source code is governed by a Apache2.0-style
|
||||||
// license that can be found in the LICENSE file.
|
// license that can be found in the LICENSE file.
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
// @Author Alan <lzh.shap@gmail.com>
|
||||||
// @Date 2023/07/25
|
// @Date 2023/07/25
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
using IP2Region.Net.Internal.Abstractions;
|
||||||
|
|
||||||
namespace IP2Region.Net.Internal;
|
namespace IP2Region.Net.Internal;
|
||||||
|
|
||||||
class VectorIndexCacheStrategy : FileCacheStrategy
|
internal class VectorIndexCacheStrategy : AbstractCacheStrategy
|
||||||
{
|
{
|
||||||
private const int VectorIndexRows = 256;
|
private readonly ReadOnlyMemory<byte> _vectorIndex;
|
||||||
private const int VectorIndexCols = 256;
|
|
||||||
|
|
||||||
private readonly ReadOnlyMemory<byte> _vectorCache;
|
|
||||||
|
|
||||||
public VectorIndexCacheStrategy(string xdbPath) : base(xdbPath)
|
public VectorIndexCacheStrategy(string xdbPath) : base(xdbPath)
|
||||||
{
|
{
|
||||||
_vectorCache = GetData(HeaderInfoLength, VectorIndexRows * VectorIndexCols * VectorIndexSize);
|
var vectorLength = VectorIndexRows * VectorIndexCols * VectorIndexSize;
|
||||||
|
_vectorIndex = base.GetData(HeaderInfoLength, vectorLength);
|
||||||
}
|
}
|
||||||
|
|
||||||
public override ReadOnlyMemory<byte> GetVectorIndex(int offset) => _vectorCache.Slice(offset, VectorIndexSize);
|
internal override ReadOnlyMemory<byte> GetVectorIndex(uint ip)
|
||||||
}
|
{
|
||||||
|
var idx = GetVectorIndexStartPos(ip);
|
||||||
|
return _vectorIndex.Slice(idx, VectorIndexSize);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -1,29 +1,17 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
|
||||||
// @Date 2023/07/25
|
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
namespace IP2Region.Net.XDB;
|
namespace IP2Region.Net.XDB;
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 缓存策略枚举
|
|
||||||
/// </summary>
|
|
||||||
public enum CachePolicy
|
public enum CachePolicy
|
||||||
{
|
{
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// no cache
|
/// no cache
|
||||||
/// </summary>
|
/// </summary>
|
||||||
File,
|
File,
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// cache vector index , reduce the number of IO operations
|
/// cache vector index , reduce the number of IO operations
|
||||||
/// </summary>
|
/// </summary>
|
||||||
VectorIndex,
|
VectorIndex,
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// default cache policy , cache whole xdb file
|
/// default cache policy , cache whole xdb file
|
||||||
/// </summary>
|
/// </summary>
|
||||||
Content
|
Content
|
||||||
}
|
}
|
||||||
|
|
@ -1,167 +1,85 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
// Copyright 2023 The Ip2Region Authors. All rights reserved.
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
// Use of this source code is governed by a Apache2.0-style
|
||||||
// license that can be found in the LICENSE file.
|
// license that can be found in the LICENSE file.
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
// @Author Alan Lee <lzh.shap@gmail.com>
|
||||||
// @Date 2023/07/25
|
// @Date 2023/07/23
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
|
using System.Net;
|
||||||
|
using System.Runtime.InteropServices;
|
||||||
|
using System.Text;
|
||||||
using IP2Region.Net.Abstractions;
|
using IP2Region.Net.Abstractions;
|
||||||
using IP2Region.Net.Internal;
|
using IP2Region.Net.Internal;
|
||||||
using System.Buffers.Binary;
|
using IP2Region.Net.Internal.Abstractions;
|
||||||
using System.Diagnostics.CodeAnalysis;
|
|
||||||
using System.Net;
|
|
||||||
using System.Text;
|
|
||||||
|
|
||||||
namespace IP2Region.Net.XDB;
|
namespace IP2Region.Net.XDB;
|
||||||
|
|
||||||
/// <summary>
|
public class Searcher : ISearcher
|
||||||
/// <see cref="ISearcher"/> 实现类
|
|
||||||
/// </summary>
|
|
||||||
/// <remarks>
|
|
||||||
/// <inheritdoc/>
|
|
||||||
/// </remarks>
|
|
||||||
public class Searcher(CachePolicy cachePolicy, string xdbPath) : ISearcher
|
|
||||||
{
|
{
|
||||||
private readonly ICacheStrategy _cacheStrategy = CacheStrategyFactory.CreateCacheStrategy(cachePolicy, xdbPath);
|
const int SegmentIndexSize = 14;
|
||||||
|
|
||||||
/// <summary>
|
private readonly AbstractCacheStrategy _cacheStrategy;
|
||||||
/// <inheritdoc/>
|
|
||||||
/// </summary>
|
|
||||||
public int IoCount => _cacheStrategy.IoCount;
|
public int IoCount => _cacheStrategy.IoCount;
|
||||||
|
|
||||||
/// <summary>
|
public Searcher(CachePolicy cachePolicy, string dbPath)
|
||||||
/// <inheritdoc/>
|
{
|
||||||
/// </summary>
|
var factory = new CacheStrategyFactory(dbPath);
|
||||||
|
_cacheStrategy = factory.CreateCacheStrategy(cachePolicy);
|
||||||
|
}
|
||||||
|
|
||||||
public string? Search(string ipStr)
|
public string? Search(string ipStr)
|
||||||
{
|
{
|
||||||
var ipAddress = IPAddress.Parse(ipStr);
|
var ip = Util.IpAddressToUInt32(ipStr);
|
||||||
return SearchCore(ipAddress.GetAddressBytes());
|
return Search(ip);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
public string? Search(IPAddress ipAddress)
|
||||||
/// <inheritdoc/>
|
|
||||||
/// </summary>
|
|
||||||
public string? Search(IPAddress ipAddress) => SearchCore(ipAddress.GetAddressBytes());
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// <inheritdoc/>
|
|
||||||
/// </summary>
|
|
||||||
[Obsolete("已弃用，请改用其他方法；Deprecated; please use Search(string) or Search(IPAddress) method.")]
|
|
||||||
[ExcludeFromCodeCoverage]
|
|
||||||
public string? Search(uint ipAddress)
|
|
||||||
{
|
{
|
||||||
var bytes = BitConverter.GetBytes(ipAddress);
|
var ip = Util.IpAddressToUInt32(ipAddress);
|
||||||
Array.Reverse(bytes);
|
return Search(ip);
|
||||||
return SearchCore(bytes);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
string? SearchCore(byte[] ipBytes)
|
public string? Search(uint ip)
|
||||||
{
|
{
|
||||||
// 重置 IO 计数器
|
var index = _cacheStrategy.GetVectorIndex(ip);
|
||||||
_cacheStrategy.ResetIoCount();
|
uint sPtr = MemoryMarshal.Read<uint>(index.Span);
|
||||||
|
uint ePtr = MemoryMarshal.Read<uint>(index.Span.Slice(4));
|
||||||
|
|
||||||
// 每个 vector 索引项的字节数
|
|
||||||
var vectorIndexSize = 8;
|
|
||||||
|
|
||||||
// vector 索引的列数
|
|
||||||
var vectorIndexCols = 256;
|
|
||||||
|
|
||||||
// 计算得到 vector 索引项的开始地址。
|
|
||||||
var il0 = ipBytes[0];
|
|
||||||
var il1 = ipBytes[1];
|
|
||||||
var idx = il0 * vectorIndexCols * vectorIndexSize + il1 * vectorIndexSize;
|
|
||||||
|
|
||||||
var vector = _cacheStrategy.GetVectorIndex(idx);
|
|
||||||
var sPtr = BinaryPrimitives.ReadUInt32LittleEndian(vector.Span);
|
|
||||||
var ePtr = BinaryPrimitives.ReadUInt32LittleEndian(vector.Span.Slice(4));
|
|
||||||
|
|
||||||
// @Note: ptr validate, zero ptr means source data missing
|
|
||||||
// so we could just stop here and return an empty string.
|
|
||||||
if (sPtr == 0 || ePtr == 0)
|
|
||||||
{
|
|
||||||
return "";
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
var length = ipBytes.Length;
|
|
||||||
var indexSize = length * 2 + 6;
|
|
||||||
var l = 0;
|
|
||||||
var h = (ePtr - sPtr) / indexSize;
|
|
||||||
var dataLen = 0;
|
var dataLen = 0;
|
||||||
long dataPtr = 0;
|
uint dataPtr = 0;
|
||||||
|
uint l = 0;
|
||||||
|
uint h = (ePtr -sPtr) / SegmentIndexSize;
|
||||||
|
|
||||||
while (l <= h)
|
while (l <= h)
|
||||||
{
|
{
|
||||||
int m = (int)(l + h) >> 1;
|
var mid = Util.GetMidIp(l, h);
|
||||||
|
var pos = sPtr + mid * SegmentIndexSize;
|
||||||
|
|
||||||
var p = sPtr + m * indexSize;
|
var buffer = _cacheStrategy.GetData((int)pos, SegmentIndexSize);
|
||||||
var buff = _cacheStrategy.GetData(p, indexSize);
|
uint sip = MemoryMarshal.Read<uint>(buffer.Span);
|
||||||
|
uint eip = MemoryMarshal.Read<uint>(buffer.Span.Slice(4));
|
||||||
|
|
||||||
var s = buff.Span.Slice(0, length);
|
if (ip < sip)
|
||||||
var e = buff.Span.Slice(length, length);
|
|
||||||
if (ByteCompare(ipBytes, s) < 0)
|
|
||||||
{
|
{
|
||||||
h = m - 1;
|
h = mid - 1;
|
||||||
}
|
}
|
||||||
else if (ByteCompare(ipBytes, e) > 0)
|
else if (ip > eip)
|
||||||
{
|
{
|
||||||
l = m + 1;
|
l = mid + 1;
|
||||||
}
|
}
|
||||||
else
|
else
|
||||||
{
|
{
|
||||||
dataLen = BinaryPrimitives.ReadUInt16LittleEndian(buff.Span.Slice(length * 2, 2));
|
dataLen = MemoryMarshal.Read<ushort>(buffer.Span.Slice(8));
|
||||||
dataPtr = BinaryPrimitives.ReadUInt32LittleEndian(buff.Span.Slice(length * 2 + 2, 4));
|
dataPtr = MemoryMarshal.Read<uint>(buffer.Span.Slice(10));
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
var regionBuff = _cacheStrategy.GetData(dataPtr, dataLen);
|
if (dataLen == 0)
|
||||||
|
{
|
||||||
|
return default;
|
||||||
|
}
|
||||||
|
|
||||||
|
var regionBuff = _cacheStrategy.GetData((int)dataPtr,dataLen);
|
||||||
return Encoding.UTF8.GetString(regionBuff.Span.ToArray());
|
return Encoding.UTF8.GetString(regionBuff.Span.ToArray());
|
||||||
}
|
}
|
||||||
|
}
|
||||||
static int ByteCompare(byte[] ip1, ReadOnlySpan<byte> ip2) => ip1.Length == 4 ? IPv4Compare(ip1, ip2) : IPv6Compare(ip1, ip2);
|
|
||||||
|
|
||||||
static int IPv4Compare(byte[] ip1, ReadOnlySpan<byte> ip2)
|
|
||||||
{
|
|
||||||
var ret = 0;
|
|
||||||
for (int i = 0; i < ip1.Length; i++)
|
|
||||||
{
|
|
||||||
var ip2Index = ip1.Length - 1 - i;
|
|
||||||
if (ip1[i] < ip2[ip2Index])
|
|
||||||
{
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
else if (ip1[i] > ip2[ip2Index])
|
|
||||||
{
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return ret;
|
|
||||||
}
|
|
||||||
|
|
||||||
static int IPv6Compare(byte[] ip1, ReadOnlySpan<byte> ip2)
|
|
||||||
{
|
|
||||||
var ret = 0;
|
|
||||||
for (int i = 0; i < ip1.Length; i++)
|
|
||||||
{
|
|
||||||
if (ip1[i] < ip2[i])
|
|
||||||
{
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
else if (ip1[i] > ip2[i])
|
|
||||||
{
|
|
||||||
return 1;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return ret;
|
|
||||||
}
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// <inheritdoc/>
|
|
||||||
/// </summary>
|
|
||||||
public void Dispose()
|
|
||||||
{
|
|
||||||
_cacheStrategy.Dispose();
|
|
||||||
GC.SuppressFinalize(this);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,20 +1,8 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
|
||||||
// @Date 2023/07/25
|
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
using System.Buffers;
|
|
||||||
using System.Buffers.Binary;
|
|
||||||
using System.Net;
|
using System.Net;
|
||||||
using System.Runtime.InteropServices;
|
using System.Runtime.InteropServices;
|
||||||
|
|
||||||
namespace IP2Region.Net.XDB;
|
namespace IP2Region.Net.XDB;
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 工具类
|
|
||||||
/// </summary>
|
|
||||||
public static class Util
|
public static class Util
|
||||||
{
|
{
|
||||||
public static uint IpAddressToUInt32(string ipAddress)
|
public static uint IpAddressToUInt32(string ipAddress)
|
||||||
|
|
@ -22,7 +10,7 @@ public static class Util
|
||||||
var address = IPAddress.Parse(ipAddress);
|
var address = IPAddress.Parse(ipAddress);
|
||||||
return IpAddressToUInt32(address);
|
return IpAddressToUInt32(address);
|
||||||
}
|
}
|
||||||
|
|
||||||
public static uint IpAddressToUInt32(IPAddress ipAddress)
|
public static uint IpAddressToUInt32(IPAddress ipAddress)
|
||||||
{
|
{
|
||||||
byte[] bytes = ipAddress.GetAddressBytes();
|
byte[] bytes = ipAddress.GetAddressBytes();
|
||||||
|
|
@ -32,60 +20,4 @@ public static class Util
|
||||||
|
|
||||||
public static uint GetMidIp(uint x, uint y)
|
public static uint GetMidIp(uint x, uint y)
|
||||||
=> (x & y) + ((x ^ y) >> 1);
|
=> (x & y) + ((x ^ y) >> 1);
|
||||||
|
}
|
||||||
public static async Task<XdbVersion> GetVersionAsync(string dbPath, CancellationToken token = default)
|
|
||||||
{
|
|
||||||
if (string.IsNullOrEmpty(dbPath))
|
|
||||||
{
|
|
||||||
throw new ArgumentNullException(nameof(dbPath));
|
|
||||||
}
|
|
||||||
|
|
||||||
if (!File.Exists(dbPath))
|
|
||||||
{
|
|
||||||
throw new FileNotFoundException("xdb file not found.", dbPath);
|
|
||||||
}
|
|
||||||
|
|
||||||
using var reader = File.OpenRead(dbPath);
|
|
||||||
return await GetVersionAsync(reader, token);
|
|
||||||
}
|
|
||||||
|
|
||||||
internal static async Task<XdbVersion> GetVersionAsync(FileStream reader, CancellationToken token = default)
|
|
||||||
{
|
|
||||||
XdbVersion ret = default;
|
|
||||||
var buffer = ArrayPool<byte>.Shared.Rent(256);
|
|
||||||
|
|
||||||
try
|
|
||||||
{
|
|
||||||
var length = await reader.ReadAsync(buffer, 0, 256, token);
|
|
||||||
if (length == 256)
|
|
||||||
{
|
|
||||||
ret = Parse(buffer);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
finally
|
|
||||||
{
|
|
||||||
ArrayPool<byte>.Shared.Return(buffer);
|
|
||||||
}
|
|
||||||
|
|
||||||
return ret;
|
|
||||||
}
|
|
||||||
|
|
||||||
private static XdbVersion Parse(ReadOnlySpan<byte> buffer)
|
|
||||||
{
|
|
||||||
var ret = new XdbVersion
|
|
||||||
{
|
|
||||||
Ver = BinaryPrimitives.ReadUInt16LittleEndian(buffer.Slice(0, 2)),
|
|
||||||
CachePolice = BinaryPrimitives.ReadUInt16LittleEndian(buffer.Slice(2, 2)),
|
|
||||||
StartIndex = BinaryPrimitives.ReadUInt32LittleEndian(buffer.Slice(8, 4)),
|
|
||||||
EndIndex = BinaryPrimitives.ReadUInt32LittleEndian(buffer.Slice(12, 4)),
|
|
||||||
IPVer = BinaryPrimitives.ReadUInt16LittleEndian(buffer.Slice(16, 2)),
|
|
||||||
BytesCount = BinaryPrimitives.ReadUInt16LittleEndian(buffer.Slice(18, 2))
|
|
||||||
};
|
|
||||||
|
|
||||||
var createdAt = BinaryPrimitives.ReadUInt32LittleEndian(buffer.Slice(4, 4));
|
|
||||||
var dtm = new DateTimeOffset(1970, 1, 1, 0, 0, 0, TimeSpan.FromHours(0));
|
|
||||||
ret.CreatedTime = dtm.AddSeconds(createdAt);
|
|
||||||
|
|
||||||
return ret;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,49 +0,0 @@
|
||||||
// Copyright 2025 The Ip2Region Authors. All rights reserved.
|
|
||||||
// Use of this source code is governed by a Apache2.0-style
|
|
||||||
// license that can be found in the LICENSE file.
|
|
||||||
// @Author Alan <lzh.shap@gmail.com>
|
|
||||||
// @Date 2023/07/25
|
|
||||||
// Updated by Argo Zhang <argo@live.ca> at 2025/11/21
|
|
||||||
|
|
||||||
namespace IP2Region.Net.XDB;
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// XdbVersion 结构体
|
|
||||||
/// </summary>
|
|
||||||
public struct XdbVersion
|
|
||||||
{
|
|
||||||
/// <summary>
|
|
||||||
/// 获得/设置 版本号
|
|
||||||
/// </summary>
|
|
||||||
public ushort Ver { get; set; }
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 获得/设置 缓存策略
|
|
||||||
/// </summary>
|
|
||||||
public ushort CachePolice { get; set; }
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 获得/设置 文件生成时间
|
|
||||||
/// </summary>
|
|
||||||
public DateTimeOffset CreatedTime { get; set; }
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 获得/设置 索引起始地址
|
|
||||||
/// </summary>
|
|
||||||
public uint StartIndex { get; set; }
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 获得/设置 索引结束地址
|
|
||||||
/// </summary>
|
|
||||||
public uint EndIndex { get; set; }
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 获得/设置 IP版本
|
|
||||||
/// </summary>
|
|
||||||
public ushort IPVer { get; set; }
|
|
||||||
|
|
||||||
/// <summary>
|
|
||||||
/// 获得/设置 指针字节数
|
|
||||||
/// </summary>
|
|
||||||
public ushort BytesCount { get; set; }
|
|
||||||
}
|
|
||||||
|
|
@ -24,71 +24,34 @@ ISearcher searcher = new Searcher(CachePolicy , "your xdb file path");
|
||||||
| CachePolicy.Content | Cache the entire `xdb` data. | Yes |
|
| CachePolicy.Content | Cache the entire `xdb` data. | Yes |
|
||||||
| CachePolicy.VectorIndex | Cache `vecotorIndex` to speed up queries and reduce system io pressure by reducing one fixed IO operation. | Yes |
|
| CachePolicy.VectorIndex | Cache `vecotorIndex` to speed up queries and reduce system io pressure by reducing one fixed IO operation. | Yes |
|
||||||
| CachePolicy.File | Completely file-based queries | Yes |
|
| CachePolicy.File | Completely file-based queries | Yes |
|
||||||
|
|
||||||
### XDB File Description
|
### XDB File Description
|
||||||
Generate using [maker](https://github.com/lionsoul2014/ip2region/tree/master/maker/csharp), or [download](https://github.com/lionsoul2014/ip2region/blob/master/data/ip2region.xdb) pre-generated xdb files
|
Generate using [maker](https://github.com/lionsoul2014/ip2region/tree/master/maker/csharp), or [download](https://github.com/lionsoul2014/ip2region/blob/master/data/ip2region.xdb) pre-generated xdb files
|
||||||
|
|
||||||
## ASP.NET Core Usage
|
## ASP.NET Core Usage
|
||||||
|
|
||||||
```csharp
|
```csharp
|
||||||
services.AddIP2RegionService("your xdb file path", cachePolicy: CachePolicy.Content);
|
services.AddSingleton<ISearcher>(new Searcher(CachePolicy , "your xdb file path"));
|
||||||
```
|
```
|
||||||
|
|
||||||
NET6/7
|
|
||||||
```csharp
|
|
||||||
provider.GetRequiredService<ISearcher>()
|
|
||||||
```
|
|
||||||
|
|
||||||
NET8+ support keyed service
|
|
||||||
```csharp
|
|
||||||
provider.GetRequiredKeyedService<ISearcher>("IP2Region.Net");
|
|
||||||
```
|
|
||||||
|
|
||||||
## TargetFrameworks
|
|
||||||
netstandard2.0;netstandard2.1;net6.0;net7.0;net8.0;net9.0;net10.0
|
|
||||||
|
|
||||||
## Performance
|
## Performance
|
||||||
// * Summary *
|
|
||||||
|
|
||||||
BenchmarkDotNet v0.15.6, Windows 11 (10.0.26200.7171)
|
``` ini
|
||||||
13th Gen Intel Core i7-13700 2.10GHz, 1 CPU, 24 logical and 16 physical cores
|
|
||||||
.NET SDK 10.0.100
|
BenchmarkDotNet=v0.13.2, OS=macOS 13.4.1 (c) (22F770820d) [Darwin 22.5.0]
|
||||||
[Host] : .NET 10.0.0 (10.0.0, 10.0.25.52411), X64 RyuJIT x86-64-v3
|
Apple M1, 1 CPU, 8 logical and 8 physical cores
|
||||||
DefaultJob : .NET 10.0.0 (10.0.0, 10.0.25.52411), X64 RyuJIT x86-64-v3
|
.NET SDK=7.0.306
|
||||||
|
[Host] : .NET 6.0.20 (6.0.2023.32017), Arm64 RyuJIT AdvSIMD
|
||||||
|
DefaultJob : .NET 6.0.20 (6.0.2023.32017), Arm64 RyuJIT AdvSIMD
|
||||||
|
|
||||||
|
|
||||||
| Method | Mean | Error | StdDev | Gen0 | Allocated |
|
```
|
||||||
|------------ |-------------:|-----------:|-----------:|-------:|----------:|
|
| Method | Mean | Error | StdDev |
|
||||||
| ContentIPv4 | 53.70 ns | 0.296 ns | 0.277 ns | 0.0086 | 136 B |
|
|-------------------------|-----------:|---------:|---------:|
|
||||||
| VectorIPv4 | 4,446.04 ns | 18.673 ns | 15.593 ns | 0.0076 | 232 B |
|
| CachePolicy_Content | 155.7 ns | 0.46 ns | 0.39 ns |
|
||||||
| FileIPv4 | 6,712.40 ns | 15.718 ns | 13.934 ns | 0.0153 | 264 B |
|
| CachePolicy_File | 2,186.8 ns | 34.27 ns | 32.06 ns |
|
||||||
| ContentIPv6 | 145.53 ns | 0.331 ns | 0.277 ns | 0.0126 | 200 B |
|
| CachePolicy_VectorIndex | 1,570.3 ns | 27.53 ns | 22.99 ns |
|
||||||
| VectorIPv6 | 7,058.39 ns | 125.505 ns | 117.398 ns | 0.0381 | 712 B |
|
|
||||||
| FileIPv6 | 10,657.97 ns | 53.907 ns | 50.425 ns | 0.0458 | 744 B |
|
|
||||||
|
|
||||||
// * Hints *
|
|
||||||
Outliers
|
|
||||||
Benchmarks.VectorIPv4: Default -> 2 outliers were removed (4.55 us, 4.58 us)
|
|
||||||
Benchmarks.FileIPv4: Default -> 1 outlier was removed (6.79 us)
|
|
||||||
Benchmarks.ContentIPv6: Default -> 2 outliers were removed (148.08 ns, 152.27 ns)
|
|
||||||
|
|
||||||
// * Legends *
|
|
||||||
Mean : Arithmetic mean of all measurements
|
|
||||||
Error : Half of 99.9% confidence interval
|
|
||||||
StdDev : Standard deviation of all measurements
|
|
||||||
Gen0 : GC Generation 0 collects per 1000 operations
|
|
||||||
Allocated : Allocated memory per single operation (managed only, inclusive, 1KB = 1024B)
|
|
||||||
1 ns : 1 Nanosecond (0.000000001 sec)
|
|
||||||
|
|
||||||
// * Diagnostic Output - MemoryDiagnoser *
|
|
||||||
|
|
||||||
|
|
||||||
// ***** BenchmarkRunner: End *****
|
|
||||||
Run time: 00:02:06 (126.09 sec), executed benchmarks: 6
|
|
||||||
|
|
||||||
Global total time: 00:02:13 (133.47 sec), executed benchmarks: 6
|
|
||||||
// * Artifacts cleanup *
|
|
||||||
Artifacts cleanup is finished
|
|
||||||
|
|
||||||
## Contributing
|
## Contributing
|
||||||
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
|
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
|
||||||
|
|
@ -96,4 +59,4 @@ Pull requests are welcome. For major changes, please open an issue first to disc
|
||||||
Please make sure to update tests as appropriate.
|
Please make sure to update tests as appropriate.
|
||||||
|
|
||||||
## License
|
## License
|
||||||
[Apache License 2.0](https://github.com/lionsoul2014/ip2region/blob/master/LICENSE.md)
|
[Apache License 2.0](https://github.com/lionsoul2014/ip2region/blob/master/LICENSE.md)
|
||||||
|
|
@ -1,18 +0,0 @@
|
||||||
.PHONY: all compile test clean bench-v4 bench-v6
|
|
||||||
|
|
||||||
all: compile
|
|
||||||
|
|
||||||
compile:
|
|
||||||
rebar3 compile
|
|
||||||
|
|
||||||
test:
|
|
||||||
rebar3 eunit
|
|
||||||
|
|
||||||
clean:
|
|
||||||
rebar3 clean
|
|
||||||
|
|
||||||
bench-v4:
|
|
||||||
sh benchmarks/xdb-benchmark.sh ipv4
|
|
||||||
|
|
||||||
bench-v6:
|
|
||||||
sh benchmarks/xdb-benchmark.sh ipv6
|
|
||||||
|
|
@ -1,180 +1,129 @@
|
||||||
:globe_with_meridians: [中文简体](README_zh.md) | [English](README.md)
|
# ip2region xdb erlang 查询客户端
|
||||||
|
|
||||||
# ip2region Erlang query client
|
### 简介
|
||||||
|
该bingding以erlang语言实现xdb查询客户端，基于Erlang OTP Application，查询逻辑由ip2region_worker工作进程实现，支持配多个工作进程来进行负载均衡。
|
||||||
|
|
||||||
### Introduction
|
### 应用配置
|
||||||
|
该应用可配置的参数在ip2region.app.src中,如下：
|
||||||
This binding implements the xdb query client in `Erlang`, based on the `Erlang OTP Application`. The query logic is implemented by the `ip2region_worker` worker process, supporting multiple worker processes for load balancing.
|
``` erlang
|
||||||
|
|
||||||
### Application Configuration
|
|
||||||
|
|
||||||
The configurable parameters for this application are in `ip2region.app.src`, as follows:
|
|
||||||
|
|
||||||
```erlang
|
|
||||||
{env,[
|
{env,[
|
||||||
{poolargs, [
|
{poolargs, [
|
||||||
{size, 1}, %% Default number of worker processes
|
{size, 1}, %% 工作进程默认数量
|
||||||
{max_overflow, 5} %% Maximum number of worker processes
|
{max_overflow, 5} %% 工作进程最大数量
|
||||||
]},
|
|
||||||
{db, [
|
|
||||||
{ipv4, "ip2region.xdb"} %% Default IPv4 xdb file
|
|
||||||
]}
|
]}
|
||||||
]}
|
]}
|
||||||
```
|
```
|
||||||
|
|
||||||
### Dual-stack configuration (IPv4 + IPv6)
|
### 编译
|
||||||
|
|
||||||
To enable IPv6 queries, add the `ipv6` entry to the `db` list and place both xdb files under `priv/`:
|
|
||||||
|
|
||||||
```erlang
|
|
||||||
{env,[
|
|
||||||
{poolargs, [
|
|
||||||
{size, 1},
|
|
||||||
{max_overflow, 5}
|
|
||||||
]},
|
|
||||||
{db, [
|
|
||||||
{ipv4, "ip2region.xdb"},
|
|
||||||
{ipv6, "ip2region_v6.xdb"}
|
|
||||||
]}
|
|
||||||
]}
|
|
||||||
```
|
```
|
||||||
|
|
||||||
The `xdb:search/1` interface automatically detects IPv4 and IPv6 inputs and routes them to the correct worker pool.
|
|
||||||
|
|
||||||
### Compile
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ rebar3 compile
|
$ rebar3 compile
|
||||||
```
|
```
|
||||||
|
### 运行
|
||||||
### Run
|
将xdb文件放到priv目录下，然后启动erlang节点：
|
||||||
|
|
||||||
Place the xdb file in the `priv` directory, then start the Erlang node:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ rebar3 shell
|
|
||||||
```
|
```
|
||||||
|
$ rebar3 shell
|
||||||
|
|
||||||
Call the `xdb:search/1` interface in the Erlang shell to query IP address information. This interface supports IP addresses represented as list strings, binary strings, tuples, and integers:
|
```
|
||||||
|
在erlang shell中调用xdb:search/1接口查询Ip地址信息, 该接口支持以list格式字符串、binary格式字符串、tuple和整数表示的IP地址，如下：
|
||||||
```erlang
|
```
|
||||||
1> xdb:search("1.0.8.0").
|
1> xdb:search("1.0.8.0").
|
||||||
[20013,22269,124,24191,19996,30465,124,24191,24030,24066,
|
[20013,22269,124,48,124,24191,19996,30465,124,24191,24030,
|
||||||
124,20013,22269,30005,20449,124,67,78]
|
24066,124,30005,20449]
|
||||||
2>
|
2>
|
||||||
3> io:format("~ts~n", [xdb:search("1.0.8.0")]).
|
3> io:format("~ts~n", [xdb:search("1.0.8.0")]).
|
||||||
中国|广东省|广州市|中国电信|CN
|
中国|0|广东省|广州市|电信
|
||||||
4> io:format("~ts~n", [xdb:search(<<"1.0.8.0">>)]).
|
io:format("~ts~n", [xdb:search(<<"1.0.8.0">>)]).
|
||||||
中国|广东省|广州市|中国电信|CN
|
中国|0|广东省|广州市|电信
|
||||||
5> io:format("~ts~n", [xdb:search({1,0,8,0})]).
|
4> io:format("~ts~n", [xdb:search({1,0,8,0})]).
|
||||||
中国|广东省|广州市|中国电信|CN
|
中国|0|广东省|广州市|电信
|
||||||
6> io:format("~ts~n", [xdb:search(16779264)]).
|
6> io:format("~ts~n", [xdb:search(16779264)]).
|
||||||
中国|广东省|广州市|中国电信|CN
|
中国|0|广东省|广州市|电信
|
||||||
```
|
```
|
||||||
|
|
||||||
With dual-stack enabled, IPv6 addresses are supported in the same way:
|
### 使用方法
|
||||||
|
* 在rebar.config中引入依赖
|
||||||
```erlang
|
|
||||||
1> io:format("~ts~n", [xdb:search("2001:4860:4860::8888")]).
|
|
||||||
United States|Florida|Miami|Google LLC|US
|
|
||||||
2> io:format("~ts~n", [xdb:search(<<"2001:4860:4860::8888">>)]).
|
|
||||||
United States|Florida|Miami|Google LLC|US
|
|
||||||
3> io:format("~ts~n", [xdb:search({8193,18528,18528,0,0,0,0,34952})]).
|
|
||||||
United States|Florida|Miami|Google LLC|US
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Usage
|
|
||||||
|
|
||||||
- Add the dependency in `rebar.config`
|
|
||||||
|
|
||||||
```erlang
|
|
||||||
{deps, [
|
{deps, [
|
||||||
ip2region
|
ip2region
|
||||||
]}.
|
]}.
|
||||||
|
|
||||||
|
```
|
||||||
|
* 启动ip2region Application
|
||||||
|
```
|
||||||
|
......
|
||||||
|
|
||||||
|
application:ensure_started(ip2region),
|
||||||
|
|
||||||
|
......
|
||||||
```
|
```
|
||||||
|
|
||||||
- Start the ip2region Application
|
* 调用xdb:search/1接口查询IP信息
|
||||||
|
```
|
||||||
|
......
|
||||||
|
|
||||||
```erlang
|
ip2region:search("1.0.8.0"),
|
||||||
{ok, _} = application:ensure_all_started(ip2region).
|
|
||||||
|
......
|
||||||
```
|
```
|
||||||
|
|
||||||
- Call the `xdb:search/1` interface to query IP information
|
### 单元测试
|
||||||
|
|
||||||
```erlang
|
|
||||||
xdb:search("1.0.8.0").
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Unit Test
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ rebar3 eunit
|
$ rebar3 eunit
|
||||||
===> Verifying dependencies...
|
===> Verifying dependencies...
|
||||||
===> Analyzing applications...
|
===> Analyzing applications...
|
||||||
===> Compiling ip2region
|
===> Compiling ip2region
|
||||||
===> Performing EUnit tests...
|
===> Performing EUnit tests...
|
||||||
=INFO REPORT==== 28-Jun-2026::04:53:28 ===
|
=INFO REPORT==== 17-Jan-2023::11:52:59.920155 ===
|
||||||
XdbFile:/Users/nana/Documents/code/ip2region/.worktrees/erlang-ipv6/binding/erlang/_build/test/lib/ip2region/priv/ip2region.xdb
|
XdbFile:/home/admin/erl-workspace/ip2region/binding/erlang/_build/test/lib/ip2region/priv/ip2region.xdb
|
||||||
|
|
||||||
....
|
....
|
||||||
Finished in 0.192 seconds
|
Finished in 0.074 seconds
|
||||||
63 tests, 0 failures
|
4 tests, 0 failures
|
||||||
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Benchmark
|
### 基准测试
|
||||||
|
```
|
||||||
Both IPv4 and IPv6 benchmarks share the same script. Run it with the desired IP version:
|
|
||||||
|
|
||||||
> `cold` = first pass over the source file: each IP triggers a real search and the
|
|
||||||
> result is written into the ETS cache. `warm` = second pass over the same list,
|
|
||||||
> where every lookup is served directly from the ETS cache.
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ cd benchmarks/
|
$ cd benchmarks/
|
||||||
$ sh xdb-benchmark.sh ipv4
|
$ sh xdb-benchmark.sh
|
||||||
```
|
===> Verifying dependencies...
|
||||||
|
===> Analyzing applications...
|
||||||
For IPv6:
|
===> Compiling ip2region
|
||||||
|
Erlang/OTP 24 [erts-12.3.2.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit]
|
||||||
```bash
|
|
||||||
$ sh xdb-benchmark.sh ipv6
|
Eshell V12.3.2.2 (abort with ^G)
|
||||||
```
|
1> =INFO REPORT==== 17-Jan-2023::11:37:35.631095 ===
|
||||||
|
XdbFile:/home/admin/erl-workspace/ip2region/binding/erlang/_build/default/lib/ip2region/priv/ip2region.xdb
|
||||||
Or use the Makefile targets from the `binding/erlang` directory:
|
|
||||||
|
===> Booted ip2region
|
||||||
```bash
|
===> Evaluating: "xdb_benchmark:main(\"../../data/ip.merge.txt\"), init:stop()."
|
||||||
$ make bench-v4
|
CPU info:
|
||||||
$ make bench-v6
|
model name : AMD EPYC 7K62 48-Core Processor
|
||||||
```
|
cache size : 512 KB
|
||||||
|
cpu MHz : 2595.124
|
||||||
#### IPv4 benchmark example
|
bogomips : 5190.24
|
||||||
|
cores/threads : 2
|
||||||
```bash
|
|
||||||
System:
|
Erlang info:
|
||||||
CPU : Apple M4
|
system_version:Erlang/OTP 24 [erts-12.3.2.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit]
|
||||||
Cores : 10 cores / 10 threads
|
load test data use 4.835593s
|
||||||
Erlang : Erlang/OTP 29 [erts-17.0.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] [dtrace]
|
|
||||||
Loaded : 487169 IPs in 1.335 s
|
start run benchmark tests
|
||||||
|
|
||||||
Benchmarks:
|
search from file:
|
||||||
cold total= 9.601s count= 487169 qps= 50740.66 avg= 0.019708 ms/op (19.708 us/op)
|
ip count:683844,
|
||||||
warm total= 0.160s count= 487169 qps= 3053164.29 avg= 0.000328 ms/op ( 0.328 us/op)
|
total time: 28.201699s,
|
||||||
|
search 24248.326315375536 times per second,
|
||||||
Done.
|
use 41.23995969841075 micro second per search
|
||||||
```
|
|
||||||
|
search from cache:
|
||||||
#### IPv6 benchmark example
|
ip count:683844,
|
||||||
|
total time: 0.671801s,
|
||||||
```bash
|
search 1017926.4395259906 times per second,
|
||||||
System:
|
use 0.9823892583688677 micro second per search
|
||||||
CPU : Apple M4
|
|
||||||
Cores : 10 cores / 10 threads
|
benchmark test finish
|
||||||
Erlang : Erlang/OTP 29 [erts-17.0.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] [dtrace]
|
|
||||||
Loaded : 638953 IPs in 2.949 s
|
|
||||||
|
|
||||||
Benchmarks:
|
|
||||||
cold total= 20.504s count= 638953 qps= 31162.52 avg= 0.032090 ms/op (32.090 us/op)
|
|
||||||
warm total= 0.444s count= 638953 qps= 1437781.56 avg= 0.000696 ms/op ( 0.696 us/op)
|
|
||||||
|
|
||||||
Done.
|
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -1,179 +0,0 @@
|
||||||
:globe_with_meridians: [中文简体](README_zh.md) | [English](README.md)
|
|
||||||
|
|
||||||
# ip2region Erlang 查询客户端
|
|
||||||
|
|
||||||
### 简介
|
|
||||||
|
|
||||||
该 bingding 以 `Erlang` 语言实现 xdb 查询客户端，基于 `Erlang OTP Application`，查询逻辑由 `ip2region_worker` 工作进程实现，支持配多个工作进程来进行负载均衡。
|
|
||||||
|
|
||||||
### 应用配置
|
|
||||||
|
|
||||||
该应用可配置的参数在 `ip2region.app.src` 中, 如下：
|
|
||||||
|
|
||||||
```erlang
|
|
||||||
{env,[
|
|
||||||
{poolargs, [
|
|
||||||
{size, 1}, %% 工作进程默认数量
|
|
||||||
{max_overflow, 5} %% 工作进程最大数量
|
|
||||||
]},
|
|
||||||
{db, [
|
|
||||||
{ipv4, "ip2region.xdb"} %% 默认 IPv4 xdb 文件
|
|
||||||
]}
|
|
||||||
]}
|
|
||||||
```
|
|
||||||
|
|
||||||
### 双栈配置（IPv4 + IPv6）
|
|
||||||
|
|
||||||
如需启用 IPv6 查询，在 `db` 列表中加入 `ipv6` 项，并将两个 xdb 文件放到 `priv/` 目录下：
|
|
||||||
|
|
||||||
```erlang
|
|
||||||
{env,[
|
|
||||||
{poolargs, [
|
|
||||||
{size, 1},
|
|
||||||
{max_overflow, 5}
|
|
||||||
]},
|
|
||||||
{db, [
|
|
||||||
{ipv4, "ip2region.xdb"},
|
|
||||||
{ipv6, "ip2region_v6.xdb"}
|
|
||||||
]}
|
|
||||||
]}
|
|
||||||
```
|
|
||||||
|
|
||||||
`xdb:search/1` 接口会自动识别 IPv4 与 IPv6 输入，并将其路由到对应的工作进程池。
|
|
||||||
|
|
||||||
### 编译
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ rebar3 compile
|
|
||||||
```
|
|
||||||
|
|
||||||
### 运行
|
|
||||||
|
|
||||||
将 xdb 文件放到 `priv` 目录下，然后启动 Erlang 节点：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ rebar3 shell
|
|
||||||
```
|
|
||||||
|
|
||||||
在 Erlang shell 中调用 `xdb:search/1` 接口查询 IP 地址信息。该接口支持 list 字符串、binary 字符串、tuple 和整数表示的 IP 地址：
|
|
||||||
|
|
||||||
```erlang
|
|
||||||
1> xdb:search("1.0.8.0").
|
|
||||||
[20013,22269,124,24191,19996,30465,124,24191,24030,24066,
|
|
||||||
124,20013,22269,30005,20449,124,67,78]
|
|
||||||
2>
|
|
||||||
3> io:format("~ts~n", [xdb:search("1.0.8.0")]).
|
|
||||||
中国|广东省|广州市|中国电信|CN
|
|
||||||
4> io:format("~ts~n", [xdb:search(<<"1.0.8.0">>)]).
|
|
||||||
中国|广东省|广州市|中国电信|CN
|
|
||||||
5> io:format("~ts~n", [xdb:search({1,0,8,0})]).
|
|
||||||
中国|广东省|广州市|中国电信|CN
|
|
||||||
6> io:format("~ts~n", [xdb:search(16779264)]).
|
|
||||||
中国|广东省|广州市|中国电信|CN
|
|
||||||
```
|
|
||||||
|
|
||||||
启用双栈后，IPv6 地址也按同样方式支持：
|
|
||||||
|
|
||||||
```erlang
|
|
||||||
1> io:format("~ts~n", [xdb:search("2001:4860:4860::8888")]).
|
|
||||||
United States|Florida|Miami|Google LLC|US
|
|
||||||
2> io:format("~ts~n", [xdb:search(<<"2001:4860:4860::8888">>)]).
|
|
||||||
United States|Florida|Miami|Google LLC|US
|
|
||||||
3> io:format("~ts~n", [xdb:search({8193,18528,18528,0,0,0,0,34952})]).
|
|
||||||
United States|Florida|Miami|Google LLC|US
|
|
||||||
```
|
|
||||||
|
|
||||||
### 使用方法
|
|
||||||
|
|
||||||
- 在 `rebar.config` 中引入依赖
|
|
||||||
|
|
||||||
```erlang
|
|
||||||
{deps, [
|
|
||||||
ip2region
|
|
||||||
]}.
|
|
||||||
```
|
|
||||||
|
|
||||||
- 启动 ip2region Application
|
|
||||||
|
|
||||||
```erlang
|
|
||||||
{ok, _} = application:ensure_all_started(ip2region).
|
|
||||||
```
|
|
||||||
|
|
||||||
- 调用 `xdb:search/1` 接口查询 IP 信息
|
|
||||||
|
|
||||||
```erlang
|
|
||||||
xdb:search("1.0.8.0").
|
|
||||||
```
|
|
||||||
|
|
||||||
### 单元测试
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ rebar3 eunit
|
|
||||||
===> Verifying dependencies...
|
|
||||||
===> Analyzing applications...
|
|
||||||
===> Compiling ip2region
|
|
||||||
===> Performing EUnit tests...
|
|
||||||
=INFO REPORT==== 28-Jun-2026::04:53:28 ===
|
|
||||||
XdbFile:/Users/nana/Documents/code/ip2region/.worktrees/erlang-ipv6/binding/erlang/_build/test/lib/ip2region/priv/ip2region.xdb
|
|
||||||
|
|
||||||
....
|
|
||||||
Finished in 0.192 seconds
|
|
||||||
63 tests, 0 failures
|
|
||||||
```
|
|
||||||
|
|
||||||
### 基准测试
|
|
||||||
|
|
||||||
IPv4 与 IPv6 共用一个脚本，通过参数指定版本：
|
|
||||||
|
|
||||||
> `cold` = 第一次遍历源文件：每个 IP 都会触发真实查询，并把结果写入 ETS 缓存。
|
|
||||||
> `warm` = 第二次遍历同一列表，所有查询都直接命中 ETS 缓存。
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ cd benchmarks/
|
|
||||||
$ sh xdb-benchmark.sh ipv4
|
|
||||||
```
|
|
||||||
|
|
||||||
IPv6：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ sh xdb-benchmark.sh ipv6
|
|
||||||
```
|
|
||||||
|
|
||||||
也可以在 `binding/erlang` 目录直接用 Makefile：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
$ make bench-v4
|
|
||||||
$ make bench-v6
|
|
||||||
```
|
|
||||||
|
|
||||||
#### IPv4 基准测试示例
|
|
||||||
|
|
||||||
```bash
|
|
||||||
System:
|
|
||||||
CPU : Apple M4
|
|
||||||
Cores : 10 cores / 10 threads
|
|
||||||
Erlang : Erlang/OTP 29 [erts-17.0.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] [dtrace]
|
|
||||||
Loaded : 487169 IPs in 1.335 s
|
|
||||||
|
|
||||||
Benchmarks:
|
|
||||||
cold total= 9.601s count= 487169 qps= 50740.66 avg= 0.019708 ms/op (19.708 us/op)
|
|
||||||
warm total= 0.160s count= 487169 qps= 3053164.29 avg= 0.000328 ms/op ( 0.328 us/op)
|
|
||||||
|
|
||||||
Done.
|
|
||||||
```
|
|
||||||
|
|
||||||
#### IPv6 基准测试示例
|
|
||||||
|
|
||||||
```bash
|
|
||||||
System:
|
|
||||||
CPU : Apple M4
|
|
||||||
Cores : 10 cores / 10 threads
|
|
||||||
Erlang : Erlang/OTP 29 [erts-17.0.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit] [dtrace]
|
|
||||||
Loaded : 638953 IPs in 2.949 s
|
|
||||||
|
|
||||||
Benchmarks:
|
|
||||||
cold total= 20.504s count= 638953 qps= 31162.52 avg= 0.032090 ms/op (32.090 us/op)
|
|
||||||
warm total= 0.444s count= 638953 qps= 1437781.56 avg= 0.000696 ms/op ( 0.696 us/op)
|
|
||||||
|
|
||||||
Done.
|
|
||||||
```
|
|
||||||
|
|
@ -1,37 +1,5 @@
|
||||||
#!/bin/bash
|
#!/bin/bash
|
||||||
set -e
|
|
||||||
|
|
||||||
# Run from binding/erlang so paths and rebar3 artifacts are correct.
|
cd ..
|
||||||
cd "$(dirname "$0")/.."
|
|
||||||
|
|
||||||
VERSION="${1:-ipv4}"
|
rebar3 shell --eval="xdb_benchmark:main(\"../../data/ip.merge.txt\"), init:stop()."
|
||||||
PROJECT_ROOT="$(cd ../.. && pwd)"
|
|
||||||
|
|
||||||
if [ "$VERSION" = "ipv6" ]; then
|
|
||||||
DB_FILE="${PROJECT_ROOT}/data/ip2region_v6.xdb"
|
|
||||||
DATA_FILE="${PROJECT_ROOT}/data/ipv6_source.txt"
|
|
||||||
DB_CONFIG="[{ipv6, \"${DB_FILE}\"}]"
|
|
||||||
elif [ "$VERSION" = "ipv4" ]; then
|
|
||||||
DATA_FILE="${PROJECT_ROOT}/data/ipv4_source.txt"
|
|
||||||
DB_CONFIG="[{ipv4, \"ip2region.xdb\"}]"
|
|
||||||
else
|
|
||||||
echo "Usage: $0 [ipv4|ipv6]"
|
|
||||||
exit 1
|
|
||||||
fi
|
|
||||||
|
|
||||||
rebar3 compile
|
|
||||||
|
|
||||||
# shellcheck disable=SC2089
|
|
||||||
EVAL="application:load(ip2region), \
|
|
||||||
application:set_env(ip2region, db, ${DB_CONFIG}), \
|
|
||||||
application:set_env(ip2region, poolargs, [{size,4},{max_overflow,8}]), \
|
|
||||||
{ok,_}=application:ensure_all_started(ip2region), \
|
|
||||||
xdb_benchmark:main(\"${DATA_FILE}\"), \
|
|
||||||
init:stop()."
|
|
||||||
|
|
||||||
# shellcheck disable=SC2090
|
|
||||||
erl -pa _build/default/lib/poolboy/ebin \
|
|
||||||
-pa _build/default/lib/ip2region/ebin \
|
|
||||||
-kernel logger_level error \
|
|
||||||
-noshell \
|
|
||||||
-eval "${EVAL}"
|
|
||||||
|
|
|
||||||
|
|
@ -4,39 +4,23 @@
|
||||||
-define(NONE, none).
|
-define(NONE, none).
|
||||||
-define(APP_NAME, ip2region).
|
-define(APP_NAME, ip2region).
|
||||||
|
|
||||||
|
-define(XDB_VECTOR_INDEX, ets_xdb_vector_index).
|
||||||
|
-define(XDB_SEGMENT_INDEX, ets_xdb_segement_index).
|
||||||
|
-define(IP2REGION_CACHE, ets_ip2region_cache).
|
||||||
|
|
||||||
|
|
||||||
-define(XDB_HEADER_SIZE, 256).
|
-define(XDB_HEADER_SIZE, 256).
|
||||||
-define(XDB_VECTOR_COLS, 256).
|
-define(XDB_VECTOR_COLS, 256).
|
||||||
-define(XDB_VECTOR_INDEX_SIZE, 8).
|
-define(XDB_VECTOR_INDEX_SIZE, 8).
|
||||||
-define(XDB_VECTOR_INDEX_COUNT, (16#10000)). %% 256*256
|
-define(XDB_VECTOR_INDEX_COUNT, (16#10000)). %% 256*256
|
||||||
|
|
||||||
%% IP versions as stored in xdb header
|
-define(XDB_SEGMENT_INDEX_SIZE, 14).
|
||||||
-define(IP_VERSION_4, 4).
|
|
||||||
-define(IP_VERSION_6, 6).
|
|
||||||
|
|
||||||
%% Segment index sizes (bytes): start_ip + end_ip + data_len + data_ptr
|
-define(IP2REGION_POOL, ip2region_pool).
|
||||||
-define(XDB_SEGMENT_INDEX_SIZE_V4, 14). %% 4 + 4 + 2 + 4
|
|
||||||
-define(XDB_SEGMENT_INDEX_SIZE_V6, 38). %% 16 + 16 + 2 + 4
|
|
||||||
|
|
||||||
%% Per-version ETS tables
|
|
||||||
-define(XDB_VECTOR_INDEX_V4, ets_xdb_vector_index_v4).
|
|
||||||
-define(XDB_VECTOR_INDEX_V6, ets_xdb_vector_index_v6).
|
|
||||||
-define(XDB_SEGMENT_INDEX_V4, ets_xdb_segment_index_v4).
|
|
||||||
-define(XDB_SEGMENT_INDEX_V6, ets_xdb_segment_index_v6).
|
|
||||||
-define(IP2REGION_CACHE_V4, ets_ip2region_cache_v4).
|
|
||||||
-define(IP2REGION_CACHE_V6, ets_ip2region_cache_v6).
|
|
||||||
|
|
||||||
%% Per-version pool names.
|
|
||||||
-define(IP2REGION_POOL_V4, ip2region_pool_v4).
|
|
||||||
-define(IP2REGION_POOL_V6, ip2region_pool_v6).
|
|
||||||
|
|
||||||
-record(xdb_header, {
|
|
||||||
version :: non_neg_integer(),
|
|
||||||
index_policy :: non_neg_integer(),
|
|
||||||
created_at :: non_neg_integer(),
|
|
||||||
start_index_ptr :: non_neg_integer(),
|
|
||||||
end_index_ptr :: non_neg_integer(),
|
|
||||||
ip_version :: non_neg_integer(),
|
|
||||||
runtime_ptr_bytes :: non_neg_integer()
|
|
||||||
}).
|
|
||||||
|
|
||||||
|
-ifndef(IF).
|
||||||
|
-define(IF(C, T, F), case (C) of true -> (T); false -> (F) end).
|
||||||
|
-define(IF(C, T), ?IF(C, T, skip)).
|
||||||
-endif.
|
-endif.
|
||||||
|
|
||||||
|
-endif.
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
{erl_opts, [
|
{erl_opts, [
|
||||||
debug_info,
|
debug_info,
|
||||||
export_all,
|
export_all,
|
||||||
nowarn_export_all
|
nowarn_export_all
|
||||||
]}.
|
]}.
|
||||||
|
|
||||||
|
|
@ -18,7 +18,7 @@
|
||||||
{ex_doc, [
|
{ex_doc, [
|
||||||
{extras, ["README.md"]},
|
{extras, ["README.md"]},
|
||||||
{main, "README.md"},
|
{main, "README.md"},
|
||||||
{source_url, "https://github.com/lionsoul2014/ip2region/tree/master/binding/erlang"}
|
{source_url, "https://github.com/leihua996/ip2region/tree/master/binding/erlang"}
|
||||||
]}.
|
]}.
|
||||||
|
|
||||||
{hex, [{doc, ex_doc}]}.
|
{hex, [{doc, ex_doc}]}.
|
||||||
|
|
|
||||||
|
|
@ -5,20 +5,16 @@
|
||||||
{mod, {ip2region_app, []}},
|
{mod, {ip2region_app, []}},
|
||||||
{applications,
|
{applications,
|
||||||
[kernel,
|
[kernel,
|
||||||
stdlib,
|
stdlib
|
||||||
poolboy
|
|
||||||
]},
|
]},
|
||||||
{env,[
|
{env,[
|
||||||
{poolargs, [
|
{poolargs, [
|
||||||
{size, 1},
|
{size, 1},
|
||||||
{max_overflow, 5}
|
{max_overflow, 5}
|
||||||
]},
|
|
||||||
{db, [
|
|
||||||
{ipv4, "ip2region.xdb"}
|
|
||||||
]}
|
]}
|
||||||
]},
|
]},
|
||||||
{modules, []},
|
{modules, []},
|
||||||
|
|
||||||
{licenses, ["Apache-2.0"]},
|
{licenses, ["Apache-2.0"]},
|
||||||
{links, [{"Github", "https://github.com/lionsoul2014/ip2region/tree/master/binding/erlang"}]}
|
{links, [{"Github", "https://github.com/leihua996/ip2region/tree/master/binding/erlang"}]}
|
||||||
]}.
|
]}.
|
||||||
|
|
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue