xinb
/
ip2region_private
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

feature: new add support ipv4 and ipv6 xdb-parse crate search in binding rust folder

This commit is contained in:
RuntimeBroker 2025-09-23 17:06:35 +08:00
parent 7576daa6cc
commit 1c0829a593
240 changed files with 1125 additions and 714760 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.gitattributes vendored
View File

@ -1,2 +0,0 @@
data/ipv6_source.txt filter=lfs diff=lfs merge=lfs -text
data/ip2region_v6.xdb filter=lfs diff=lfs merge=lfs -text

92
.gitignore vendored
View File

@ -1,92 +0,0 @@
*.class
*.out
*.o
*.pyc
*~
*.log
*.la
*.so
*.xdb
*.iml
META-INF/
.DS_Store
# Binary Files #
*.jar
!dbMaker-*.jar
# ignore all xdb except the one in ./data/
*.xdb
!/data/*.xdb
# Package Files #
.settings/
.classpath
.project
# vim swp file #
*.swp
.idea
.vscode
# binding
/v1.0//binding/java/classes/
/v1.0//binding/java/doc/
/v1.0//binding/java/target/
/v1.0//binding/java/*.jar
/binding/java/classes/
/binding/java/doc/
/binding/java/target/
/binding/java/*.jar
# clang
/binding/c/xdb_searcher
/binding/c/util_test
/binding/c/cmake-build-debug
# lua/luc_c
/binding/lua_c/cmake-build-debug
# golang
/binding/golang/searcher
/binding/golang/xdb_searcher
/binding/golang/golang
# rust
Cargo.lock
target
# VS ignore cases
/**/*.sln
/v1.0/binding/c#/**/.vs/
/v1.0/binding/c#/**/packages
/v1.0/binding/c#/**/bin
/v1.0/binding/c#/**/obj
/binding/c#/**/.vs/
/binding/c#/**/packages
/binding/c#/**/bin
/binding/c#/**/obj
# Nodejs
/v1.0/binding/nodejs/tests/unitTests/__snapshots__
/v1.0/binding/nodejs/coverage
/binding/nodejs/tests/unitTests/__snapshots__
/binding/nodejs/coverage
/binding/nodejs/node_modules
/binding/nodejs/.nyc_output
/binging/nodejs/package-lock.json
# maker
## golang
/maker/golang/dbmaker
/maker/golang/xdb_maker
/maker/golang/golang
#erlang
/binding/erlang/_build
/binding/erlang/doc
#vscode
.vscode
build

225
LICENSE.md
View File

@ -1,225 +0,0 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
1. Definitions.
"License" shall mean the terms and conditions for use, reproduction,
and distribution as defined by Sections 1 through 9 of this document.
"Licensor" shall mean the copyright owner or entity authorized by
the copyright owner that is granting the License.
"Legal Entity" shall mean the union of the acting entity and all
other entities that control, are controlled by, or are under common
control with that entity. For the purposes of this definition,
"control" means (i) the power, direct or indirect, to cause the
direction or management of such entity, whether by contract or
otherwise, or (ii) ownership of fifty percent (50%) or more of the
outstanding shares, or (iii) beneficial ownership of such entity.
"You" (or "Your") shall mean an individual or Legal Entity
exercising permissions granted by this License.
"Source" form shall mean the preferred form for making modifications,
including but not limited to software source code, documentation
source, and configuration files.
"Object" form shall mean any form resulting from mechanical
transformation or translation of a Source form, including but
not limited to compiled object code, generated documentation,
and conversions to other media types.
"Work" shall mean the work of authorship, whether in Source or
Object form, made available under the License, as indicated by a
copyright notice that is included in or attached to the work
(an example is provided in the Appendix below).
"Derivative Works" shall mean any work, whether in Source or Object
form, that is based on (or derived from) the Work and for which the
editorial revisions, annotations, elaborations, or other modifications
represent, as a whole, an original work of authorship. For the purposes
of this License, Derivative Works shall not include works that remain
separable from, or merely link (or bind by name) to the interfaces of,
the Work and Derivative Works thereof.
"Contribution" shall mean any work of authorship, including
the original version of the Work and any modifications or additions
to that Work or Derivative Works thereof, that is intentionally
submitted to Licensor for inclusion in the Work by the copyright owner
or by an individual or Legal Entity authorized to submit on behalf of
the copyright owner. For the purposes of this definition, "submitted"
means any form of electronic, verbal, or written communication sent
to the Licensor or its representatives, including but not limited to
communication on electronic mailing lists, source code control systems,
and issue tracking systems that are managed by, or on behalf of, the
Licensor for the purpose of discussing and improving the Work, but
excluding communication that is conspicuously marked or otherwise
designated in writing by the copyright owner as "Not a Contribution."
"Contributor" shall mean Licensor and any individual or Legal Entity
on behalf of whom a Contribution has been received by Licensor and
subsequently incorporated within the Work.
2. Grant of Copyright License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
copyright license to reproduce, prepare Derivative Works of,
publicly display, publicly perform, sublicense, and distribute the
Work and such Derivative Works in Source or Object form.
3. Grant of Patent License. Subject to the terms and conditions of
this License, each Contributor hereby grants to You a perpetual,
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
(except as stated in this section) patent license to make, have made,
use, offer to sell, sell, import, and otherwise transfer the Work,
where such license applies only to those patent claims licensable
by such Contributor that are necessarily infringed by their
Contribution(s) alone or by combination of their Contribution(s)
with the Work to which such Contribution(s) was submitted. If You
institute patent litigation against any entity (including a
cross-claim or counterclaim in a lawsuit) alleging that the Work
or a Contribution incorporated within the Work constitutes direct
or contributory patent infringement, then any patent licenses
granted to You under this License for that Work shall terminate
as of the date such litigation is filed.
4. Redistribution. You may reproduce and distribute copies of the
Work or Derivative Works thereof in any medium, with or without
modifications, and in Source or Object form, provided that You
meet the following conditions:
(a) You must give any other recipients of the Work or
Derivative Works a copy of this License; and
(b) You must cause any modified files to carry prominent notices
stating that You changed the files; and
(c) You must retain, in the Source form of any Derivative Works
that You distribute, all copyright, patent, trademark, and
attribution notices from the Source form of the Work,
excluding those notices that do not pertain to any part of
the Derivative Works; and
(d) If the Work includes a "NOTICE" text file as part of its
distribution, then any Derivative Works that You distribute must
include a readable copy of the attribution notices contained
within such NOTICE file, excluding those notices that do not
pertain to any part of the Derivative Works, in at least one
of the following places: within a NOTICE text file distributed
as part of the Derivative Works; within the Source form or
documentation, if provided along with the Derivative Works; or,
within a display generated by the Derivative Works, if and
wherever such third-party notices normally appear. The contents
of the NOTICE file are for informational purposes only and
do not modify the License. You may add Your own attribution
notices within Derivative Works that You distribute, alongside
or as an addendum to the NOTICE text from the Work, provided
that such additional attribution notices cannot be construed
as modifying the License.
You may add Your own copyright statement to Your modifications and
may provide additional or different license terms and conditions
for use, reproduction, or distribution of Your modifications, or
for any such Derivative Works as a whole, provided Your use,
reproduction, and distribution of the Work otherwise complies with
the conditions stated in this License.
5. Submission of Contributions. Unless You explicitly state otherwise,
any Contribution intentionally submitted for inclusion in the Work
by You to the Licensor shall be under the terms and conditions of
this License, without any additional terms or conditions.
Notwithstanding the above, nothing herein shall supersede or modify
the terms of any separate license agreement you may have executed
with Licensor regarding such Contributions.
6. Trademarks. This License does not grant permission to use the trade
names, trademarks, service marks, or product names of the Licensor,
except as required for reasonable and customary use in describing the
origin of the Work and reproducing the content of the NOTICE file.
7. Disclaimer of Warranty. Unless required by applicable law or
agreed to in writing, Licensor provides the Work (and each
Contributor provides its Contributions) on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied, including, without limitation, any warranties or conditions
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
PARTICULAR PURPOSE. You are solely responsible for determining the
appropriateness of using or redistributing the Work and assume any
risks associated with Your exercise of permissions under this License.
8. Limitation of Liability. In no event and under no legal theory,
whether in tort (including negligence), contract, or otherwise,
unless required by applicable law (such as deliberate and grossly
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
result of this License or out of the use or inability to use the
Work (including but not limited to damages for loss of goodwill,
work stoppage, computer failure or malfunction, or any and all
other commercial damages or losses), even if such Contributor
has been advised of the possibility of such damages.
9. Accepting Warranty or Additional Liability. While redistributing
the Work or Derivative Works thereof, You may choose to offer,
and charge a fee for, acceptance of support, warranty, indemnity,
or other liability obligations and/or rights consistent with this
License. However, in accepting such obligations, You may act only
on Your own behalf and on Your sole responsibility, not on behalf
of any other Contributor, and only if You agree to indemnify,
defend, and hold each Contributor harmless for any liability
incurred by, or claims asserted against, such Contributor by reason
of your accepting any such warranty or additional liability.
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright [yyyy] [name of copyright owner]
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
==========================================================================
The following license applies to the ip2region library
--------------------------------------------------------------------------
Copyright (c) 2015 Lionsoul<chenxin619315@gmail.com>
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

115
ReadMe.md
View File

@ -1,115 +0,0 @@
# Ip2region 是什么
[ip2region](https://ip2region.net) - 是一个离线IP地址定位库和IP定位数据管理框架，同时支持`IPv4`和`IPv6`，10微秒级别的查询效率，提供了众多主流编程语言的 `xdb` 数据生成和查询客户端实现。
# Ip2region 特性
### 1、离线定位库
项目本身同时了提供了一份 IPv4(`data/ipv4_source.txt`) 和 IPv6(`data/ipv6_source.txt`) 的原始数据和对应的 xdb 文件(`data/ip2region_v4.xdb` 和 `data/ip2region_v6.xdb`) 用于实现精确到城市的的查询定位功能。
### 2、数据管理框架
`xdb` 支持亿级别的 IP 数据段行数，region 信息支持完全自定义，自带数据的 region 信息固定了格式为：`国家|省份|城市|ISP`，你可以在 region 中追加特定业务需求的数据，例如：GPS信息/国际统一地域信息编码/邮编等。也就是你完全可以使用 ip2region 来管理你自己的 IP 定位数据。
### 3、数据去重和压缩
`xdb` 格式生成程序会自动处理输入的原始数据，检查并且完成相连 IP 段的的合并以及相同地域信息的去重和压缩。
### 4、极速查询响应
即使是完全基于 `xdb` 文件的查询，单次查询响应时间在十微秒级别，可通过如下两种方式开启内存加速查询：
1. `vIndex` 索引缓存 ：使用固定的 `512KiB` 的内存空间缓存 vector index 数据，减少一次 IO 磁盘操作，保持平均查询效率稳定在100微秒之内。
2. `xdb` 整个文件缓存：将整个 `xdb` 文件全部加载到内存，内存占用等同于 `xdb` 文件大小，无磁盘 IO 操作，保持10微秒级别的查询效率。
### 5、统一的查询接口
`xdb` 提供了版本兼容的查询实现，一个统一的 API 可以同时提供对 IPv4 和 IPv6 数据的查询并且返回统一的数据。
# `xdb` 数据查询
API 介绍，使用文档和测试程序请参考对应 `searcher` 查询客户端下的 ReadMe 介绍，全部查询 binding 实现情况如下：
| 编程语言 | 描述 | IPv4 支持 | IPv6 支持 | 贡献者 |
|:---------------------------------|:----------------------------|:-------------------|:------------------ |:--------------------------------------------------|
| [Golang](binding/golang) | golang xdb 查询客户端 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
| [PHP](binding/php) | php xdb 查询客户端 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
| [Java](binding/java) | java xdb 查询客户端 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
| [C](binding/c) | POSIX C xdb 查询客户端 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
| [Lua_c](binding/lua_c) | lua c 扩展 xdb 查询客户端 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
| [Lua](binding/lua) | lua xdb 查询客户端 | :white_check_mark: | :x: | [Lion](https://github.com/lionsoul2014) |
| [Rust](binding/rust) | rust xdb 查询客户端 | :white_check_mark: | :x: | [gongzhengyang](https://github.com/gongzhengyang) |
| [Python](binding/python) | python xdb 查询客户端 | :white_check_mark: | :x: | [厉害的花花](https://github.com/luckydog6132) |
| [Nodejs](binding/nodejs) | nodejs xdb 查询客户端 | :white_check_mark: | :x: | [Wu Jian Ping](https://github.com/wujjpp) |
| [Csharp](binding/csharp) | csharp xdb 查询客户端 | :white_check_mark: | :x: | [Alen Lee](https://github.com/malus2077) |
| [Erlang](binding/erlang) | erlang xdb 查询客户端 | :white_check_mark: | :x: | [malou](https://github.com/malou996) |
| [Nginx](binding/nginx) | nginx 扩展 xdb 查询客户端 | :white_check_mark: | :x: | [Wu Jian Ping](https://github.com/wujjpp) |
| [C++](binding/cpp) | C++ xdb 查询客户端 | :white_check_mark: | :x: | [Yunbin Liu](https://github.com/liuyunbin) |
| [Typescript](binding/typescript) | Typescript xdb 查询客户端 | :white_check_mark: | :x: | [Alan Lee](https://github.com/malus2077) |
以下工具链实现由社区开发者通过第三方仓库贡献：
| 编程语言 | 描述 | 贡献者 |
|:--------------------------------------------------------------- |:------------------------|:-----------------------------------------------|
| [ip2region-composer](https://github.com/zoujingli/ip2region) | php composer 管理客户端 | [邹景立](https://github.com/zoujingli) |
| [ruby-ip2region](https://github.com/jicheng1014/ruby-ip2region) | ruby xdb 查询客户端实现 | [jicheng1014](https://github.com/jicheng1014) |
| [Ip2regionTool](https://github.com/orestonce/Ip2regionTool) | ip2region 数据转换工具 | [orestonce](https://github.com/orestonce) |
# `xdb` 数据生成
API 介绍，使用文档和测试程序请参考如下 `maker` 生成程序下的 ReadMe 文档：
| 编程语言 | 描述 | IPv4 支持 | IPv6 支持 | 贡献者 |
| :---------------------- | :------------------- |:-------------------|:-------------------| :----------------------------------------- |
| [Golang](maker/golang) | golang xdb 生成程序 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
| [Java](maker/java) | java xdb 生成程序 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
| [Python](maker/python) | python xdb 生成程序 | :white_check_mark: | :x: | [leolin49](https://github.com/leolin49) |
| [Csharp](maker/csharp) | csharp xdb 生成程序 | :white_check_mark: | :x: | [Alan Lee](https://github.com/malus2077) |
| [Rust](maker/rust) | rust xdb 生成程序 | :white_check_mark: | :x: | [KevinWang](https://github.com/KevinWL) |
| [C++](maker/cpp) | C++ xdb 生成程序 | :white_check_mark: | :x: | [Yunbin Liu](https://github.com/liuyunbin) |
# `xdb` 数据更新
ip2region 项目的核心在于 <b>研究 IP 数据的存储和快速查询的设计和实现</b>， 项目自带的 `./data/ipv4_source.txt` 和 `./data/ipv6_source.txt` 原始数据不会再提供更新，对于数据精度和更新频率要求很高的使用场景建议到 [Ip2Region社区](https://ip2region.net/products/offline) 或者第三方购买商用离线数据，你可以使用如下几种方式来尝试自己更新数据：
### 手动编辑更新
你可以基于 ip2region 自带的 `./data/ipv4_source.txt` 和 `./data/ipv6_source.txt` 原始 IP 数据用 ip2region 提供的编辑工具来自己修改，目前数据源有如下几种方式：
1. ip2region 社区提供的数据（请参考地底部的公众号关注社区通知）
2. ip2region Github/Gitee 中带有 `[数据源补充]` 标签的 Issue
3. 其他自定义数据：例如客户提供的数据，或者通过 GPS 和 WIFI 定位得到的数据，或者来自其他平台的合法合规的数据
原始 IP 数据编辑工具使用方法请参考如下的 `maker` 生成程序下的 ReadMe 文档：
| 编程语言 | 描述 | IPv4 支持 | IPv6 支持 | 贡献者 |
|:------------------------------------|:-------------------------|:-------------------|:-------------------|:-------------------------------------------|
| [Golang](maker/golang#xdb-数据编辑) | golang IP 原始数据编辑器 | :white_check_mark: | :white_check_mark: | [Lion](https://github.com/lionsoul2014) |
| [Java](maker/java#xdb-数据编辑) | java IP 原始数据编辑器 | :white_check_mark: | :soon: | [Lion](https://github.com/lionsoul2014) |
| [C++](maker/cpp#xdb-数据编辑) | C++ IP 原始数据编辑器 | :white_check_mark: | :x: | [Yunbin Liu](https://github.com/liuyunbin) |
### 检测自动更新
如果你想通过你自己的 API 或数据源来更新数据，你可以参考以下视频分享的 `基于检测算法` 的更新算法来自己编写一个更新程序：
1. [数据更新实现视频分享 - part1](https://www.bilibili.com/video/BV1934y1E7Q5/)
2. [数据更新实现视频分享 - part2](https://www.bilibili.com/video/BV1pF411j7Aw/)
# 官方社区
Ip2Region 官方社区正式上线于 `2025/06/12` 日，一方面提供了稳定的 [商用离线数据](https://ip2region.net/products/offline) 服务，另一方面便于在核心代码外强化 IP 工具链和数据服务，例如 [使用文档](https://ip2region.net/doc/)，[查询测试](https://ip2region.net/search/demo)，数据纠错等，更多关于社区的信息和服务请访问 [Ip2Region 官方社区](https://ip2region.net/)。
# 相关备注
### 1、并发查询必读
xdb 整个缓存的查询都 <b>是</b> 并发安全的，基于文件的查询都 <b>不是</b> 并发安全的实现，不同进程/线程/协程需要通过创建不同的查询对象来安全使用，并发量很大的情况下，基于文件查询的方式可能会导致打开文件数过多的错误，请修改内核的最大允许打开文件数(fs.file-max=一个更高的值)，或者将整个xdb加载到内存进行安全并发使用。
### 2、技术资源分享
1. xdb 数据结构分析：[“ip2region xdb-数据结构描述“](https://ip2region.net/doc/xdb/ipv4_structure)
2. xdb 查询过程分析：[“ip2region xdb-查询过程描述”](https://ip2region.net/doc/xdb/ipv4_search)
3. xdb 生成过程分析：[“ip2region xdb-生成过程描述”](https://ip2region.net/doc/xdb/ipv4_generate)
4. xdb 文件生成教程：[“ip2region xdb-文件生成教程”](https://ip2region.net/doc/data/ipv4_xdb_make)
5. xdb 数据更新方法：[“ip2region 数据更新和 xdb 数据编辑器的使用”](https://mp.weixin.qq.com/s/cZH5qIn4E5rQFy6N32RCzA)
### 3、技术信息博客
1. [Ip2Region 官方社区](https://ip2region.net)
2. 请先关注微信公众号 lionsoul-org (狮子的魂)

27
binding/c/Makefile
View File

@ -1,27 +0,0 @@
all: xdb_searcher test_util
xdb_searcher: xdb_api.h xdb_util.c xdb_searcher.c main.c
gcc -O2 -I./ xdb_util.c xdb_searcher.c main.c -o xdb_searcher
test_util: xdb_api.h xdb_util.c test_util.c
gcc -O2 -I./ xdb_util.c test_util.c -o test_util
xdb_searcher.o: xdb_searcher.c
gcc -c xdb_searcher.c
xdb_util.o: xdb_util.c
gcc -c xdb_util.c
xdb_searcher_lib: xdb_util.o xdb_searcher.o
mkdir -p build/lib
mkdir -p build/include
ar -rc build/lib/libxdb.a `find . -name *.o`
cp xdb_api.h build/include
clean:
find ./ -name \*.o | xargs rm -f
find ./ -name test_util | xargs rm -f
find ./ -name xdb_searcher | xargs rm -f
rm -rf build
.PHONY: all clean xdb_searcher test_util

341
binding/c/ReadMe.md
View File

@ -1,341 +0,0 @@
# ip2region xdb c 查询客户端实现
# 使用方式
### 关于 IPv4 和 IPv6
该 xdb 查询客户端实现同时支持对 IPv4 和 IPv6 的查询，使用方式如下：
```c
#include "xdb_api.h";
// 如果是 IPv4: 设置 xdb 路径为 v4 的 xdb 文件，IP版本指定为 IPv4
const char *db_path = "../../data/ip2region_v4.xdb"; // 或者你的 ipv4 xdb 的路径
xdb_version_t *version = XDB_IPv4;
// 如果是 IPv6: 设置 xdb 路径为 v6 的 xdb 文件，IP版本指定为 IPv6
const char *db_path = "../../data/ip2region_v6.xdb"; // 或者你的 ipv6 xdb 路径
xdb_version_t *version = XDB_IPv6;
// db_path 指定的 xdb 的 IP 版本必须和 version 指定的一致，不然查询执行的时候会报错
// 备注：以下演示直接使用 db_path 和 version 变量
```
### XDB 文件验证
建议您主动去验证 xdb 文件的适用性，因为后期的一些新功能可能会导致目前的 Searcher 版本无法适用你使用的 xdb 文件，验证可以避免运行过程中的一些不可预测的错误。 你不需要每次都去验证，例如在服务启动的时候，或者手动调用命令验证确认版本匹配即可，不要在每次创建的 Searcher 的时候运行验证，这样会影响查询的响应速度，尤其是高并发的使用场景。
```c
#include "xdb_api.h";
int errcode = xdb_verify(db_path);
if ($err != 0) {
// 适用性验证失败！！！
// 当前查询客户端实现不适用于 db_path 指定的 xdb 文件的查询.
// 应该停止启动服务，使用合适的 xdb 文件或者升级到适合 db_path 的 Searcher 实现。
printf("failed to verify xdb file `%s`, errcode: %d\n", db_path, errcode);
return;
}
// 验证通过，当前使用的 Searcher 可以安全的用于对 dbPath 指向的 xdb 的查询操作
```
### 完全基于文件的查询
```c
#include <stdio.h>
#include "xdb_api.h"
int main(int argc, char *argv[]) {
xdb_searcher_t searcher;
char region_buffer[512] = {'\0'};
xdb_region_buffer_t region;
// 使用栈空间的 region_buffer 初始化 region_buffer_t
int err = xdb_region_buffer_init(&region, region_buffer, sizeof(region_buffer));
if (err != 0) {
printf("failed to init the region buffer with errcode=%d\n", err);
return 1;
}
// 在服务启动的时候初始化 winsock，不需要重复调用，只需要在 windows 系统下调用
err = xdb_init_winsock();
if (err != 0) {
printf("failed to init the winsock with errno=%d\n", err);
return 1;
}
// 1、从 db_path 初始化 xdb 查询对象.
// @Note: 使用顶部描述的 db_path 和 version 来创建 searcher
err = xdb_new_with_file_only(version, &searcher, db_path);
if (err != 0) {
printf("failed to create xdb searcher from `%s` with errno=%d\n", db_path, err);
return 1;
}
// 2、调用 search API 查询，IPv4 和 IPv6 都支持.
const char *ip_string = "1.2.3.4";
// ip_string = "2001:4:112:ffff:ffff:ffff:ffff:ffff"; // IPv6
long cost_time = 0, s_time = xdb_now();
err = xdb_search_by_string(&searcher, ip_string, &region);
cost_time = (int) (xdb_now() - s_time);
if (err != 0) {
printf("failed search(%s) with errno=%d\n", ip_string, err);
} else {
printf("{region: %s, took: %d μs}", region.value, cost_time);
}
// 清理 region 信息的内存资源，每次 search 之后都得调用
xdb_region_buffer_free(&region);
// 备注：并发使用，每一个线程需要单独定义并且初始化一个 searcher 查询对象。
// 3、关闭 xdb 查询器
xdb_close(&searcher);
xdb_clean_winsock(); // windows 下调用
return 0;
}
```
### 缓存 `VectorIndex` 索引
我们可以提前从 xdb 文件中加载出来 VectorIndex 数据，然后全局缓存，每次创建 Searcher 对象的时候使用全局的 VectorIndex 缓存可以减少一次固定的 IO 操作，从而加速查询，减少 IO 压力。
```c
#include <stdio.h>
#include "xdb_api.h"
int main(int argc, char *argv[]) {
xdb_vector_index_t *v_index;
xdb_searcher_t searcher;
xdb_region_buffer_t region;
// 使用 NULL 初始化 region_buffer，让其自动管理内存的分配
int err = xdb_region_buffer_init(&region, NULL, 0);
if (err != 0) {
printf("failed to init the region buffer with errcode=%d\n", err);
return 0;
}
// 在服务启动的时候初始化 winsock，不需要重复调用，只需要在 windows 系统下调用
err = xdb_init_winsock();
if (err != 0) {
printf("failed to init the winsock with errno=%d\n", err);
return 1;
}
// 1、从顶部描述的 db_path 加载 VectorIndex 索引。
// 得到 v_index 做成全局缓存，便于后续反复使用。
// 注意：v_index 不需要每次都加载，建议在服务启动的时候加载一次，然后做成全局资源。
v_index = xdb_load_vector_index_from_file(db_path);
if (v_index == NULL) {
printf("failed to load vector index from `%s`\n", db_path);
return 1;
}
// 2、使用全局的 VectorIndex 变量创建带 VectorIndex 缓存的 xdb 查询对象.
// @Note: 使用顶部描述的 db_path 和 version 来创建 searcher
err = xdb_new_with_vector_index(version, &searcher, db_path, v_index);
if (err != 0) {
printf("failed to create vector index cached searcher with errcode=%d\n", err);
return 2;
}
// 3、调用 search API 查询，IPv4 和 IPv6 都支持
const char *ip_string = "1.2.3.4";
// ip_string = "2001:4:112:ffff:ffff:ffff:ffff:ffff"; // IPv6
long cost_time = 0, s_time = xdb_now();
err = xdb_search_by_string(&searcher, ip_string, &region);
cost_time = (int) (xdb_now() - s_time);
if (err != 0) {
printf("failed search(%s) with errno=%d\n", ip_string, err);
} else {
printf("{region: %s, took: %d μs}", region.value, cost_time);
}
// 清理 region 信息的内存资源，每次 search 之后都得调用
xdb_region_buffer_free(&region);
// 备注：并发使用，每一个线程需要单独定义并且初始化一个 searcher 查询对象。
// 4、关闭 xdb 查询器，如果是要关闭服务，也需要释放 v_index 的内存。
xdb_close(&searcher);
xdb_close_vector_index(v_index);
xdb_clean_winsock();
return 0;
}
```
### 缓存整个 `xdb` 数据
我们也可以预先加载整个 xdb 文件到内存，然后基于这个数据创建查询对象来实现完全基于内存的查询，类似之前的 memory search。
```c
#include <stdio.h>
#include "xdb_api.h"
int main(int argc, char *argv[]) {
xdb_content_t *c_buffer;
xdb_searcher_t searcher;
xdb_region_buffer_t region;
// 使用 NULL 初始化 region_buffer，让其自动管理内存的分配
int err = xdb_region_buffer_init(&region, NULL, 0);
if (err != 0) {
printf("failed to init the region buffer with errcode=%d\n", err);
return 0;
}
// 在服务启动的时候初始化 winsock，不需要重复调用，只需要在 windows 系统下调用
err = xdb_init_winsock();
if (err != 0) {
printf("failed to init the winsock with errno=%d\n", err);
return 1;
}
// 1、从 顶部描述的 db_path 加载整个 xdb 的数据。
c_buffer = xdb_load_content_from_file(db_path);
if (v_index == NULL) {
printf("failed to load xdb content from `%s`\n", db_path);
return 1;
}
// 2、使用全局的 c_buffer 变量创建一个完全基于内存的 xdb 查询对象.
// @Note: 使用顶部描述的 version 来创建 searcher.
err = xdb_new_with_buffer(version, &searcher, c_buffer);
if (err != 0) {
printf("failed to create content cached searcher with errcode=%d\n", err);
return 2;
}
// 3、调用 search API 查询，IPv4 和 IPv6 都支持
const char *ip_string = "1.2.3.4";
// ip_string = "2001:4:112:ffff:ffff:ffff:ffff:ffff"; // IPv6
long cost_time = 0, s_time = xdb_now();
err = xdb_search_by_string(&searcher, ip_string, &region);
cost_time = (int) (xdb_now() - s_time);
if (err != 0) {
printf("failed search(%s) with errno=%d\n", ip_string, err);
} else {
printf("{region: %s, took: %d μs}", region.value, cost_time);
}
// 清理 region 信息的内存资源，每次 search 之后都得调用
xdb_region_buffer_free(&region);
// 备注：并发使用，使用这种方式创建的 xdb 查询对象可以安全用于并发。
// 建议在服务启动的时候创建好，然后一直安全并发使用，直到服务关闭。
// 4、关闭 xdb 查询器，关闭服务的时候需要释放 c_buffer 的内存。
xdb_close(&searcher);
xdb_close_content(c_buffer);
xdb_clean_winsock();
return 0;
}
```
### 关于定位信息的存储
在旧版本的实现中，search相关的函数都是依靠指定一个 `region_buffer` 内存来用于存储地域信息，这种方式还是有很大的局限性。
新的实现提供了一个 `xdb_region_buffer_t` 对象来管理这些内存的分配，你依然可以指定一个固定的 `region_buffer` 来创建 region 的内存管理，这个情况适合当你的地域信息的最大长度是可知的，这种方式可以减少运行过程中内存的碎片堆积。如果地域信息的长度不确定或者你的程序不适合提前分配一块内存来管理，你可以通过指定 NULL 的方式来初始化 `xdb_region_buffer_t`，这样对象会自动管理内存的分配，也适合任意长度的地域信息的存储，不过这种方式在长期的运行过程中肯定会增加内存碎片的堆积。
```c
// 1, 通过指定一块内存来创建 region_buffer
char buffer[512];
xdb_region_buffer_t region;
int err = xdb_region_buffer_init(&region, buffer, sizeof(buffer));
if (err != 0) {
// 初始化失败
printf("failed to init region buffer width errcode=%d", err);
return;
}
// 2，通过指定 NULL 来创建 region_buffer，让其自动按需分配内存
xdb_region_buffer_t region;
int err = xdb_region_buffer_init(&region, NULL, 0);
if (err != 0) {
// 初始化失败
printf("failed to init region buffer width errcode=%d", err);
return;
}
// 备注：在每次调用 search 完成 IP 定位信息的查询后，你需要手动调用函数来释放内存 .
// search 函数使用未经清理的 region 信息会报错。
xdb_region_buffer_free(&region);
```
# 测试程序编译
通过如下方式编译得到 xdb_searcher 可执行程序：
```bash
# cd 到 c binding 根目录
➜ c git:(fr_c_ipv6) ✗ make
gcc -O2 -I./ xdb_util.c xdb_searcher.c main.c -o xdb_searcher
gcc -O2 -I./ xdb_util.c test_util.c -o test_util
```
# 查询测试
通过 `xdb_searcher search` 命令来测试对 xdb 的查询：
```bash
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher search
./xdb_searcher search [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：使用默认的 data/ip2region_v4.xdb 进行 IPv4 查询测试：
```bash
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher search --db=../../data/ip2region_v4.xdb
ip2region xdb searcher test program
source xdb: ../../data/ip2region_v4.xdb (IPv4, vectorIndex)
type 'quit' to exit
ip2region>> 120.229.45.2
{region: 中国|广东省|深圳市|移动, io_count: 3, took: 29 μs}
```
例如：使用默认的 data/ip2region_v6.xdb 进行 IPv6 查询测试：
```bash
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher search --db=../../data/ip2region_v6.xdb
ip2region xdb searcher test program
source xdb: ../../data/ip2region_v6.xdb (IPv6, vectorIndex)
type 'quit' to exit
ip2region>> ::
{region: |||, io_count: 2, took: 38 μs}
ip2region>> 2604:bc80:8001:11a4:ffff:ffff:ffff:ffff
{region: 中国|广东省|深圳市|数据中心, io_count: 13, took: 77 μs}
```
输入 ip 即可进行查询，输入 quit 即可退出测试程序。也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同的缓存实现的效率。
# bench 测试
通过 `xdb_searcher bench` 命令来进行 bench 测试，一方面确保查询程序和 `xdb` 文件没有错误，另一方面可以通过大量的查询得到评价的查询性能：
```bash
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher bench
./xdb_searcher bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：通过默认的 data/ip2region_v4.xdb 和 data/ipv4_source.txt 来进行 IPv4 的 bench 测试：
```bash
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher bench --db=../../data/ip2region_v4.xdb --src=../../data/ipv4_source.txt
Bench finished, {cache_policy: vectorIndex, total: 1367686, took: 7.640s, cost: 5 μs/op}
```
例如：通过默认的 data/ip2region_v6.xdb 和 data/ipv6_source.txt 来进行 IPv6 的 bench 测试：
```bash
➜ c git:(fr_c_ipv6) ✗ ./xdb_searcher bench --db=../../data/ip2region_v6.xdb --src=../../data/ipv6_source.txt
Bench finished, {cache_policy: vectorIndex, total: 34159862, took: 857.750s, cost: 24 μs/op}
```
可以设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 不同缓存实现机制的效率。 @Note：请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件相同的源文件。

428
binding/c/main.c
View File

@ -1,428 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/28
#include "stdio.h"
#include "xdb_api.h"
struct searcher_test_entry {
xdb_searcher_t searcher;
xdb_vector_index_t *v_index;
xdb_content_t *c_buffer;
// xdb region buffer
char region_buffer[256];
xdb_region_buffer_t region;
};
typedef struct searcher_test_entry searcher_test_t;
int init_searcher_test(searcher_test_t *test, char *db_path, char *cache_policy) {
int err, errcode = 0;
FILE *handle = fopen(db_path, "rb");
if (handle == NULL) {
return -1;
}
// auto detect the version from the xdb header
xdb_header_t *header = xdb_load_header(handle);
if (header == NULL) {
printf("failed to load header from `%s`\n", db_path);
errcode = 1;
goto defer;
}
// verify the current xdb
err = xdb_verify_from_header(handle, header);
if (err != 0) {
printf("failed to verify xdb file `%s` with errno=%d\n", db_path, err);
errcode = 2;
goto defer;
}
xdb_version_t *version = xdb_version_from_header(header);
if (version == NULL) {
printf("failed to load version from header\n");
errcode = 3;
goto defer;
}
test->v_index = NULL;
test->c_buffer = NULL;
if (strcmp(cache_policy, "file") == 0) {
err = xdb_new_with_file_only(version, &test->searcher, db_path);
if (err != 0) {
printf("failed to create searcher with errcode=%d\n", err);
errcode = 4;
goto defer;
}
} else if (strcmp(cache_policy, "vectorIndex") == 0) {
test->v_index = xdb_load_vector_index_from_file(db_path);
if (test->v_index == NULL) {
printf("failed to load vector index from `%s`\n", db_path);
errcode = 4;
goto defer;
}
err = xdb_new_with_vector_index(version, &test->searcher, db_path, test->v_index);
if (err != 0) {
printf("failed to create vector index cached searcher with errcode=%d\n", err);
errcode = 5;
goto defer;
}
} else if (strcmp(cache_policy, "content") == 0) {
test->c_buffer = xdb_load_content_from_file(db_path);
if (test->c_buffer == NULL) {
printf("failed to load xdb content from `%s`\n", db_path);
errcode = 4;
goto defer;
}
err = xdb_new_with_buffer(version, &test->searcher, test->c_buffer);
if (err != 0) {
printf("failed to create content cached searcher with errcode=%d\n", err);
errcode = 5;
goto defer;
}
} else {
printf("invalid cache policy `%s`, options: file/vectorIndex/content\n", cache_policy);
errcode = 6;
goto defer;
}
// init the region buffer
err = xdb_region_buffer_init(&test->region, test->region_buffer, sizeof(test->region_buffer));
if (err != 0) {
printf("failed to init the region buffer with err=%d\n", err);
errcode = 7;
goto defer;
}
defer:
if (header != NULL) {
xdb_free_header(header);
}
if (handle != NULL) {
fclose(handle);
}
return errcode;
}
void destroy_searcher_test(searcher_test_t *test) {
xdb_close(&test->searcher);
// check and free the vector index
if (test->v_index != NULL) {
xdb_free_vector_index(test->v_index);
test->v_index = NULL;
}
// check and free the content buffer
if (test->c_buffer != NULL) {
xdb_free_content(test->c_buffer);
test->c_buffer = NULL;
}
}
//read a line from a command line.
static char *get_line(FILE *fp, char *__dst) {
register int c;
register char *cs;
cs = __dst;
while ( ( c = getc( fp ) ) != EOF ) {
if ( c == '\n' ) break;
*cs++ = c;
}
*cs = '\0';
return ( c == EOF && cs == __dst ) ? NULL : __dst;
}
void print_help(char *argv[]) {
printf("ip2region xdb searcher\n");
printf("%s [command] [command options]\n", argv[0]);
printf("Command: \n");
printf(" search search input test\n");
printf(" bench search bench test\n");
}
void test_search(int argc, char *argv[]) {
int i, n, err;
// for args parse
char *r, key[33] = {'\0'}, val[256] = {'\0'};
char db_file[256] = {'\0'}, cache_policy[16] = {"vectorIndex"};
// for search
long s_time, c_time;
unsigned int ip;
char line[512] = {'\0'}, region[512] = {'\0'};
// ip parse
xdb_version_t *version;
bytes_ip_t ip_bytes[16] = {'\0'};
searcher_test_t test;
for (i = 2; i < argc; i++) {
r = argv[i];
if (strlen(r) < 5) {
continue;
}
if (r[0] != '-' || r[1] != '-') {
continue;
}
if (strchr(r, '=') == NULL) {
printf("missing = for args pair '%s'\n", r);
return;
}
n = sscanf(r+2, "%32[^=]=%255[^\n]", key, val);
if (n != 2) {
printf("invalid option flag `%s`\n", r);
return;
}
// printf("key=%s, val=%s\n", key, val);
if (strcmp(key, "db") == 0) {
snprintf(db_file, sizeof(db_file), "%s", val);
} else if (strcmp(key, "cache-policy") == 0) {
memcpy(cache_policy, val, sizeof(cache_policy) - 1);
// snprintf(cache_policy, sizeof(cache_policy), "%s", val);
} else {
printf("undefined option `%s`\n", r);
return;
}
}
if (strlen(db_file) < 1) {
printf("%s search [command options]\n", argv[0]);
printf("options:\n");
printf(" --db string ip2region binary xdb file path\n");
printf(" --cache-policy string cache policy: file/vectorIndex/content\n");
return;
}
// init the win sock
err = xdb_init_winsock();
if (err != 0) {
printf("failed to init the winsock with errno=%d\n", err);
return;
}
// printf("db_file=%s, cache_policy=%s\n", db_file, cache_policy);
err = init_searcher_test(&test, db_file, cache_policy);
if (err != 0) {
// init program will print the error reasons;
return;
}
printf("ip2region xdb searcher test program\n"
"source xdb: %s (%s, %s)\n"
"type 'quit' to exit\n", db_file, xdb_get_version(&test.searcher)->name, cache_policy);
while ( 1 ) {
printf("ip2region>> ");
get_line(stdin, line);
if ( strlen(line) < 2 ) {
continue;
}
if (strcasecmp( line, "quit") == 0 ) {
break;
}
version = xdb_parse_ip(line, ip_bytes, sizeof(ip_bytes));
if (version == NULL) {
printf("invalid ip address `%s`\n", line);
continue;
}
s_time = xdb_now();
err = xdb_search(&test.searcher, ip_bytes, version->bytes, &test.region);
if (err != 0) {
printf("{err: %d, io_count: %d}\n", err, xdb_get_io_count(&test.searcher));
} else {
c_time = xdb_now() - s_time;
printf("{region: %s, io_count: %d, took: %ld μs}\n", test.region.value, xdb_get_io_count(&test.searcher), c_time);
}
// free the region
xdb_region_buffer_free(&test.region);
}
destroy_searcher_test(&test);
xdb_clean_winsock();
printf("searcher test program exited, thanks for trying\n");
}
void test_bench(int argc, char *argv[]) {
int i, n, err;
char *r, key[33] = {'\0'}, val[256] = {'\0'};
char db_file[256] = {'\0'}, src_file[256] = {'\0'}, cache_policy[16] = {"vectorIndex"};
FILE *handle;
char line[1024] = {'\0'}, sip_str[INET6_ADDRSTRLEN+1] = {'\0'}, eip_str[INET6_ADDRSTRLEN+1] = {'\0'};
char src_region[512] = {'\0'};
int count = 0, took;
long s_time, t_time, c_time = 0;
// ip parse
xdb_version_t *s_version, *e_version;
bytes_ip_t sip_bytes[16] = {'\0'}, eip_bytes[16] = {'\0'};
string_ip_t ip_string[INET6_ADDRSTRLEN] = {'\0'};
bytes_ip_t *ip_list[2];
searcher_test_t test;
for (i = 2; i < argc; i++) {
r = argv[i];
if (strlen(r) < 5) {
continue;
}
if (r[0] != '-' || r[1] != '-') {
continue;
}
if (strchr(r, '=') == NULL) {
printf("missing = for args pair '%s'\n", r);
return;
}
n = sscanf(r+2, "%32[^=]=%255[^\n]", key, val);
if (n != 2) {
printf("invalid option flag `%s`\n", r);
return;
}
if (strcmp(key, "db") == 0) {
snprintf(db_file, sizeof(db_file), "%s", val);
} else if (strcmp(key, "src") == 0) {
snprintf(src_file, sizeof(src_file), "%s", val);
} else if (strcmp(key, "cache-policy") == 0) {
memcpy(cache_policy, val, sizeof(cache_policy) - 1);
} else {
printf("undefined option `%s`\n", r);
return;
}
}
if (strlen(db_file) < 1 || strlen(src_file) < 1) {
printf("%s bench [command options]\n", argv[0]);
printf("options:\n");
printf(" --db string ip2region binary xdb file path\n");
printf(" --src string source ip text file path\n");
printf(" --cache-policy string cache policy: file/vectorIndex/content\n");
return;
}
// init the win sock
err = xdb_init_winsock();
if (err != 0) {
printf("failed to init the winsock with errno=%d\n", err);
return;
}
// printf("db_file=%s, src_file=%s, cache_policy=%s\n", db_file, src_file, cache_policy);
s_time = xdb_now();
err = init_searcher_test(&test, db_file, cache_policy);
if (err != 0) {
// the init function will print the details;
return;
}
// open the source file
handle = fopen(src_file, "r");
if (handle == NULL) {
printf("failed to open source text file `%s`\n", src_file);
return;
}
while(fgets(line, sizeof(line), handle) != NULL) {
n = sscanf(line, "%46[^|]|%46[^|]|%511[^\n]", sip_str, eip_str, src_region);
if (n != 3) {
printf("invalid ip segment line `%s`\n", line);
return;
}
s_version = xdb_parse_ip(sip_str, sip_bytes, sizeof(sip_bytes));
if (s_version == NULL) {
printf("invalid start ip `%s`\n", sip_str);
return;
}
e_version = xdb_parse_ip(eip_str, eip_bytes, sizeof(eip_bytes));
if (e_version == NULL) {
printf("invalid end ip `%s`\n", sip_str);
return;
}
if (s_version->id != e_version->id) {
printf("start ip and end ip version not match for line `%s`\n", line);
return;
}
if (xdb_ip_sub_compare(sip_bytes, s_version->bytes, eip_bytes, 0) > 0) {
printf("start ip(%s) should not be greater than end ip(%s)\n", sip_str, eip_str);
return;
}
ip_list[0] = sip_bytes;
ip_list[1] = eip_bytes;
for (i = 0; i < 2; i++) {
t_time = xdb_now();
err = xdb_search(&test.searcher, ip_list[i], s_version->bytes, &test.region);
c_time += xdb_now() - t_time;
if (err != 0) {
xdb_ip_to_string(ip_list[i], s_version->bytes, ip_string, sizeof(ip_string));
printf("failed to search ip `%s` with errno=%d\n", ip_string, err);
return;
}
// check the region info
if (strcmp(test.region.value, src_region) != 0) {
xdb_ip_to_string(ip_list[i], s_version->bytes, ip_string, sizeof(ip_string));
printf("failed to search(%s) with (%s != %s)\n", ip_string, test.region.value, src_region);
return;
}
// free the region buffer
xdb_region_buffer_free(&test.region);
count++;
}
};
took = xdb_now() - s_time;
destroy_searcher_test(&test);
xdb_clean_winsock();
fclose(handle);
printf("Bench finished, {cache_policy: %s, total: %d, took: %.3fs, cost: %d μs/op}\n",
cache_policy, count, took/1e6, count == 0 ? 0 : (int)(c_time/count));
}
int main(int argc, char *argv[]) {
if (argc < 2) {
print_help(argv);
return 0;
}
char *opt = argv[1];
if (strcmp(opt, "search") == 0) {
test_search(argc, argv);
} else if (strcmp(opt, "bench") == 0) {
test_bench(argc, argv);
} else {
print_help(argv);
}
return 0;
}

BIN
binding/c/test_util
View File

Binary file not shown.

213
binding/c/test_util.c
View File

@ -1,213 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/27
#include "stdio.h"
#include "xdb_api.h"
typedef void (* test_func_ptr) ();
struct test_func_entry {
char *name;
test_func_ptr func;
};
typedef struct test_func_entry test_func_t;
void test_load_header() {
xdb_header_t *header = xdb_load_header_from_file("../../data/ip2region_v4.xdb");
if (header == NULL) {
printf("failed to load header");
} else {
printf("header loaded: {\n"
" version: %d, \n"
" index_policy: %d, \n"
" created_at: %u, \n"
" start_index_ptr: %d, \n"
" end_index_ptr: %d\n"
" ip_version: %d\n"
" runtime_ptr_bytes: %d\n"
" length: %d\n"
"}\n",
header->version, header->index_policy, header->created_at,
header->start_index_ptr, header->end_index_ptr,
header->ip_version, header->runtime_ptr_bytes, header->length
);
}
xdb_free_header(header);
}
void test_load_vector_index() {
xdb_vector_index_t *v_index = xdb_load_vector_index_from_file("../../data/ip2region_v4.xdb");
if (v_index == NULL) {
printf("failed to load vector index from file\n");
} else {
printf("vector index loaded from file, length=%d\n", v_index->length);
}
xdb_free_vector_index(v_index);
}
void test_load_content() {
xdb_content_t *content = xdb_load_content_from_file("../../data/ip2region_v4.xdb");
if (content == NULL) {
printf("failed to load content from file\n");
} else {
printf("content loaded from file, length=%d\n", content->length);
}
xdb_free_content(content);
}
void test_parse_ip() {
const char *ip_list[] = {
"1.0.0.0", "58.251.30.115", "192.168.1.100",
"::", "2c0f:fff0::", "2fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff", "240e:982:e617:ffff:ffff:ffff:ffff:ffff",
"219.xx.xx.11", "::xx:ffff",
NULL
};
int errcode;
xdb_version_t *version;
bytes_ip_t ip_bytes[16] = {'\0'};
string_ip_t ip_string[INET6_ADDRSTRLEN] = {'\0'};
// init the sock env (for windows)
if ((errcode = xdb_init_winsock()) != 0) {
printf("failed to init the winsock");
return;
}
for (int i = 0;; i++) {
if (ip_list[i] == NULL) {
break;
}
version = xdb_parse_ip(ip_list[i], ip_bytes, sizeof(ip_bytes));
if (version == NULL) {
printf("failed to parse ip `%s`\n", ip_list[i]);
continue;
}
xdb_ip_to_string(ip_bytes, version->bytes, ip_string, sizeof(ip_string));
printf("ip: %s (version=v%d), toString: %s\n", ip_list[i], version->id, ip_string);
}
// clean up the winsock
xdb_clean_winsock();
}
struct ip_pair {
const char *sip;
const char *eip;
};
void test_ip_compare() {
struct ip_pair ip_pair_list[] = {
{"1.0.0.0", "1.0.0.1"},
{"192.168.1.101", "192.168.1.90"},
{"219.133.111.87", "114.114.114.114"},
{"1.0.4.0", "1.0.1.0"},
{"1.0.4.0", "1.0.3.255"},
{"2000::", "2000:ffff:ffff:ffff:ffff:ffff:ffff:ffff"},
{"2001:4:112::", "2001:4:112:ffff:ffff:ffff:ffff:ffff"},
{"ffff::", "2001:4:ffff:ffff:ffff:ffff:ffff:ffff"},
{NULL, NULL}
};
struct ip_pair *pair_ptr = NULL;
bytes_ip_t sip_bytes[16] = {'\0'};
bytes_ip_t eip_bytes[16] = {'\0'};
xdb_version_t *s_version, *e_version;
int bytes, errcode;
// init the sock env (for windows)
if ((errcode = xdb_init_winsock()) != 0) {
printf("failed to init the winsock");
return;
}
for (int i = 0; ;i++) {
pair_ptr = &ip_pair_list[i];
if (pair_ptr->sip == NULL) {
break;
}
s_version = xdb_parse_ip(pair_ptr->sip, sip_bytes, sizeof(sip_bytes));
if (s_version == NULL) {
printf("failed to parse sip `%s`", pair_ptr->sip);
continue;
}
e_version = xdb_parse_ip(pair_ptr->eip, eip_bytes, sizeof(eip_bytes));
if (e_version == NULL) {
printf("failed to parse eip `%s`", pair_ptr->eip);
continue;
}
if (s_version->id != e_version->id) {
printf("sip and eip version not match `%s` != `%s`\n", s_version->name, e_version->name);
continue;
}
printf(
"ip_sub_compare(%s, %s): %d\n",
pair_ptr->sip, pair_ptr->eip,
xdb_ip_sub_compare(sip_bytes, s_version->bytes, eip_bytes, 0)
);
}
// clean up the winsock
xdb_clean_winsock();
}
// please register your function heare
static test_func_t _test_function_list[] = {
// xdb buffer
{"test_load_header", test_load_header},
{"test_load_vector_index", test_load_vector_index},
{"test_load_content", test_load_content},
// ip utils
{"test_parse_ip", test_parse_ip},
{"test_ip_compare", test_ip_compare},
{NULL, NULL}
};
// valgrind --tool=memcheck --leak-check=full ./a.out
int main(int argc, char *argv[]) {
int i;
char *name;
// check and call the function
if (argc < 2) {
printf("please specified the function name to call\n");
return 1;
}
name = argv[1];
test_func_ptr func = NULL;
for (i = 0; ; i++) {
if (_test_function_list[i].name == NULL) {
break;
}
if (strcmp(name, _test_function_list[i].name) == 0) {
func = _test_function_list[i].func;
break;
}
}
if (func == NULL) {
printf("can't find test function `%s`\n", name);
return 1;
}
// call the function
func();
return 0;
}

268
binding/c/xdb_api.h
View File

@ -1,268 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/27
#ifndef C_IP2REGION_XDB_H
#define C_IP2REGION_XDB_H
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#if ( defined(WIN32) || defined(_WIN32) || defined(__WINDOWS_) || defined(WINNT) )
# define XDB_PUBLIC(type) extern __declspec(dllexport) type
# define XDB_PRIVATE(type) static type
# define XDB_WINDOWS
#include <windows.h>
#include <winsock2.h>
#include <ws2tcpip.h>
#pragma comment(lib, "ws2_32.lib")
#elif ( defined(linux) || defined(_UNIX) || defined(__APPLE__) )
# define XDB_PUBLIC(type) extern type
# define XDB_PRIVATE(type) static inline type
# define XDB_LINUX
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#endif
#define xdb_calloc( _blocks, _bytes ) calloc( _blocks, _bytes )
#define xdb_malloc( _bytes ) malloc( _bytes )
#define xdb_free( _ptr ) free( _ptr )
// public constants define
#define xdb_structure_20 2
#define xdb_structure_30 3
#define xdb_header_info_length 256
#define xdb_vector_index_rows 256
#define xdb_vector_index_cols 256
#define xdb_vector_index_size 8
#define xdb_segment_index_size 14
// --- ip version info
#define xdb_ipv4_id 4
#define xdb_ipv6_id 6
#define xdb_ipv4_bytes 4
#define xdb_ipv6_bytes 16
// cache of vector_index_row × vector_index_rows × vector_index_size
#define xdb_vector_index_length 524288
// --- xdb buffer functions
// use the following buffer struct to wrap the binary buffer data
// since the buffer data could not be operated with the string API.
struct xdb_header {
unsigned short version;
unsigned short index_policy;
unsigned int created_at;
unsigned int start_index_ptr;
unsigned int end_index_ptr;
// since 3.0+ with IPv6 supporting
unsigned short ip_version;
unsigned short runtime_ptr_bytes;
// the original buffer
unsigned int length;
char buffer[xdb_header_info_length];
};
typedef struct xdb_header xdb_header_t;
XDB_PUBLIC(xdb_header_t *) xdb_load_header(FILE *);
XDB_PUBLIC(xdb_header_t *) xdb_load_header_from_file(const char *);
XDB_PUBLIC(void) xdb_free_header(void *);
// --- vector index buffer
struct xdb_vector_index {
unsigned int length;
char buffer[xdb_vector_index_length];
};
typedef struct xdb_vector_index xdb_vector_index_t;
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index(FILE *);
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index_from_file(const char *);
XDB_PUBLIC(void) xdb_free_vector_index(void *);
// --- content buffer
struct xdb_content {
unsigned int length;
char *buffer;
};
typedef struct xdb_content xdb_content_t;
XDB_PUBLIC(xdb_content_t *) xdb_load_content(FILE *);
XDB_PUBLIC(xdb_content_t *) xdb_load_content_from_file(const char *);
XDB_PUBLIC(void) xdb_free_content(void *);
// --- xdb verify
// Verify if the current Searcher could be used to search the specified xdb file.
// Why do we need this check ?
// The future features of the xdb impl may cause the current searcher not able to work properly.
//
// @Note: You Just need to check this ONCE when the service starts
// Or use another process (eg, A command) to check once Just to confirm the suitability.
XDB_PUBLIC(int) xdb_verify(FILE *);
XDB_PUBLIC(int) xdb_verify_from_header(FILE *handle, xdb_header_t *);
XDB_PUBLIC(int) xdb_verify_from_file(const char *);
// --- End xdb buffer
// types type define
typedef char string_ip_t;
typedef unsigned char bytes_ip_t;
// --- ip version
#define XDB_IPv4 (xdb_version_v4())
#define XDB_IPv6 (xdb_version_v6())
typedef int (* ip_compare_fn_t) (const bytes_ip_t *, int, const char *, int);
struct xdb_ip_version_entry {
int id; // version id
char *name; // version name
int bytes; // ip bytes number
int segment_index_size; // segment index size in bytes
// function to compare two ips
ip_compare_fn_t ip_compare;
};
typedef struct xdb_ip_version_entry xdb_version_t;
XDB_PUBLIC(xdb_version_t *) xdb_version_v4();
XDB_PUBLIC(xdb_version_t *) xdb_version_v6();
XDB_PUBLIC(int) xdb_version_is_v4(const xdb_version_t *);
XDB_PUBLIC(int) xdb_version_is_v6(const xdb_version_t *);
XDB_PUBLIC(xdb_version_t *) xdb_version_from_name(char *);
XDB_PUBLIC(xdb_version_t *) xdb_version_from_header(xdb_header_t *);
// --- END ip version
// --- xdb util functions
// to compatiable with the windows
// returns: 0 for ok and -1 for failed
XDB_PUBLIC(int) xdb_init_winsock();
XDB_PUBLIC(void) xdb_clean_winsock();
// get the current time in microseconds
XDB_PUBLIC(long) xdb_now();
// get unsigned long (4bytes) from a specified buffer start from the specified offset with little-endian
XDB_PUBLIC(unsigned int) xdb_le_get_uint32(const char *, int);
// get unsigned short (2bytes) from a specified buffer start from the specified offset with little-endian
XDB_PUBLIC(int) xdb_le_get_uint16(const char *, int);
// parse the specified IP address to byte array.
// returns: xdb_version_t for valid ipv4 / ipv6, or NULL for failed
XDB_PUBLIC(xdb_version_t *) xdb_parse_ip(const string_ip_t *, bytes_ip_t *, size_t);
// parse the specified IPv4 address to byte array
// returns: xdb_version_t for valid ipv4, or NULL for failed
XDB_PUBLIC(xdb_version_t *) xdb_parse_v4_ip(const string_ip_t *, bytes_ip_t *, size_t);
// parse the specified IPv6 address to byte array
// returns: xdb_version_t for valid ipv6, or NULL for failed
XDB_PUBLIC(xdb_version_t *) xdb_parse_v6_ip(const string_ip_t *, bytes_ip_t *, size_t);
// convert a specified ip bytes to humen-readable string.
// returns: 0 for success or -1 for failed.
XDB_PUBLIC(int) xdb_ip_to_string(const bytes_ip_t *, int, char *, size_t);
// ipv4 bytes to string
XDB_PUBLIC(int) xdb_v4_ip_to_string(const bytes_ip_t *, char *, size_t);
// ipv6 bytes to string
XDB_PUBLIC(int) xdb_v6_ip_to_string(const bytes_ip_t *, char *, size_t);
// compare the specified ip bytes with another ip bytes in the specified buff from offset.
// ip args must be the return value from #xdb_parse_ip.
// returns: -1 if ip1 < ip2, 1 if ip1 > ip2 or 0
XDB_PUBLIC(int) xdb_ip_sub_compare(const bytes_ip_t *, int, const char *, int);
// --- END xdb utils
// --- xdb searcher api
// xdb region info structure
#define xdb_region_buffer_wrapper 1
#define xdb_region_buffer_auto 2
struct xdb_region_buffer_entry {
int type; // buffer type
char *value; // region value
size_t length; // buffer length
};
typedef struct xdb_region_buffer_entry xdb_region_buffer_t;
// wrapper the region from a local stack buffer.
// returns: 0 for succeed or failed
XDB_PUBLIC(int) xdb_region_buffer_init(xdb_region_buffer_t *, char *, size_t);
// do the buffer alloc.
// returns: 0 for ok or failed
XDB_PUBLIC(int) xdb_region_buffer_alloc(xdb_region_buffer_t *, int);
XDB_PUBLIC(void) xdb_region_buffer_free(xdb_region_buffer_t *);
// xdb searcher structure
struct xdb_searcher_entry {
// ip version
xdb_version_t *version;
// xdb file handle
FILE *handle;
// header info
const char *header;
int io_count;
// vector index buffer cache.
// preload the vector index will reduce the number of IO operations
// thus speedup the search process.
const xdb_vector_index_t *v_index;
// content buffer.
// cache the whole xdb content.
const xdb_content_t *content;
};
typedef struct xdb_searcher_entry xdb_searcher_t;
// xdb searcher new api define
XDB_PUBLIC(int) xdb_new_with_file_only(xdb_version_t *, xdb_searcher_t *, const char *);
XDB_PUBLIC(int) xdb_new_with_vector_index(xdb_version_t *, xdb_searcher_t *, const char *, const xdb_vector_index_t *);
XDB_PUBLIC(int) xdb_new_with_buffer(xdb_version_t *, xdb_searcher_t *, const xdb_content_t *);
XDB_PUBLIC(void) xdb_close(void *);
// xdb searcher search api define
XDB_PUBLIC(int) xdb_search_by_string(xdb_searcher_t *, const string_ip_t *, xdb_region_buffer_t *);
XDB_PUBLIC(int) xdb_search(xdb_searcher_t *, const bytes_ip_t *, int, xdb_region_buffer_t *);
XDB_PUBLIC(xdb_version_t *) xdb_get_version(xdb_searcher_t *);
XDB_PUBLIC(int) xdb_get_io_count(xdb_searcher_t *);
// --- END xdb searcher api
#endif // C_IP2REGION_XDB_H

257
binding/c/xdb_searcher.c
View File

@ -1,257 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/27
#include "xdb_api.h"
// --- region buffer
XDB_PUBLIC(int) xdb_region_buffer_init(xdb_region_buffer_t *region, char *buffer, size_t length) {
if (buffer == NULL) {
region->type = xdb_region_buffer_auto;
region->length = 0;
} else if (length <= 0) {
return 1;
} else {
region->type = xdb_region_buffer_wrapper;
region->length = length;
memset(buffer, 0x00, length); // zero-fill the buffer
}
region->value = buffer;
return 0;
}
XDB_PUBLIC(int) xdb_region_buffer_alloc(xdb_region_buffer_t *region, int length) {
if (length <= 0) {
return 1;
}
// no allocation supports for the buffer wapper
if (region->type == xdb_region_buffer_wrapper) {
if (length >= region->length) {
return 2;
}
region->value[length] = '\0';
return 0;
}
// ensure that the value were freed
// by calling #xdb_region_buffer_free
if (region->value != NULL) {
return 3;
}
char *ptr = (char *) xdb_malloc(length + 1);
if (ptr == NULL) {
return 4;
}
ptr[length] = '\0'; // NULL-end
region->value = ptr;
region->length = length;
return 0;
}
XDB_PUBLIC(void) xdb_region_buffer_free(xdb_region_buffer_t *region) {
if (region->type == xdb_region_buffer_auto) {
xdb_free(region->value);
region->value = NULL;
}
}
// --- END region buffer
// internal function prototype define
XDB_PRIVATE(int) read(xdb_searcher_t *, long offset, char *, size_t length);
XDB_PRIVATE(int) xdb_new_base(xdb_version_t *version, xdb_searcher_t *xdb, const char *db_path, const xdb_vector_index_t *v_index, const xdb_content_t *c_buffer) {
memset(xdb, 0x00, sizeof(xdb_searcher_t));
// set the version
xdb->version = version;
// check the content buffer first
if (c_buffer != NULL) {
xdb->v_index = NULL;
xdb->content = c_buffer;
return 0;
}
// open the xdb binary file
FILE *handle = fopen(db_path, "rb");
if (handle == NULL) {
return 1;
}
xdb->handle = handle;
xdb->v_index = v_index;
return 0;
}
// xdb searcher new api define
XDB_PUBLIC(int) xdb_new_with_file_only(xdb_version_t *version, xdb_searcher_t *xdb, const char *db_path) {
return xdb_new_base(version, xdb, db_path, NULL, NULL);
}
XDB_PUBLIC(int) xdb_new_with_vector_index(xdb_version_t *version, xdb_searcher_t *xdb, const char *db_path, const xdb_vector_index_t *v_index) {
return xdb_new_base(version, xdb, db_path, v_index, NULL);
}
XDB_PUBLIC(int) xdb_new_with_buffer(xdb_version_t *version, xdb_searcher_t *xdb, const xdb_content_t *c_buffer) {
return xdb_new_base(version, xdb, NULL, NULL, c_buffer);
}
XDB_PUBLIC(void) xdb_close(void *ptr) {
xdb_searcher_t *xdb = (xdb_searcher_t *) ptr;
if (xdb->handle != NULL) {
fclose(xdb->handle);
xdb->handle = NULL;
}
}
// --- xdb searcher search api define
XDB_PUBLIC(int) xdb_search_by_string(xdb_searcher_t *xdb, const string_ip_t *ip_string, xdb_region_buffer_t *region) {
bytes_ip_t ip_bytes[16] = {'\0'};
xdb_version_t *version = xdb_parse_ip(ip_string, ip_bytes, sizeof(ip_bytes));
if (version == NULL) {
return 10;
} else {
return xdb_search(xdb, ip_bytes, version->bytes, region);
}
}
XDB_PUBLIC(int) xdb_search(xdb_searcher_t *xdb, const bytes_ip_t *ip_bytes, int ip_len, xdb_region_buffer_t *region) {
int il0, il1, idx, err, bytes, d_bytes;
register int seg_index_size, l, h, m, p;
unsigned int s_ptr, e_ptr, data_ptr, data_len;
char vector_buffer[xdb_vector_index_size];
char *segment_buffer = NULL;
// ip version check
if (ip_len != xdb->version->bytes) {
return -1;
}
// some resets
err = 0;
data_len = 0;
bytes = xdb->version->bytes;
d_bytes = xdb->version->bytes << 1;
xdb->io_count = 0;
// locate the segment index block based on the vector index
il0 = (int) (ip_bytes[0]);
il1 = (int) (ip_bytes[1]);
idx = il0 * xdb_vector_index_cols * xdb_vector_index_size + il1 * xdb_vector_index_size;
if (xdb->v_index != NULL) {
s_ptr = xdb_le_get_uint32(xdb->v_index->buffer, idx);
e_ptr = xdb_le_get_uint32(xdb->v_index->buffer, idx + 4);
} else if (xdb->content != NULL) {
s_ptr = xdb_le_get_uint32(xdb->content->buffer, xdb_header_info_length + idx);
e_ptr = xdb_le_get_uint32(xdb->content->buffer, xdb_header_info_length + idx + 4);
} else {
err = read(xdb, xdb_header_info_length + idx, vector_buffer, sizeof(vector_buffer));
if (err != 0) {
return 10 + err;
}
s_ptr = xdb_le_get_uint32(vector_buffer, 0);
e_ptr = xdb_le_get_uint32(vector_buffer, 4);
}
// printf("s_ptr=%u, e_ptr=%u\n", s_ptr, e_ptr);
// binary search to get the final region info
seg_index_size = xdb->version->segment_index_size;
segment_buffer = xdb_malloc(seg_index_size);
if (segment_buffer == NULL) {
return -2;
}
data_len = 0, data_ptr = 0;
l = 0, h = ((int) (e_ptr - s_ptr)) / seg_index_size;
while (l <= h) {
m = (l + h) >> 1;
p = s_ptr + m * seg_index_size;
// read the segment index item
err = read(xdb, p, segment_buffer, seg_index_size);
if (err != 0) {
err += 20;
goto defer;
}
// decode the data fields as needed
if (xdb->version->ip_compare(ip_bytes, bytes, segment_buffer, 0) < 0) {
h = m - 1;
} else if (xdb->version->ip_compare(ip_bytes, bytes, segment_buffer, bytes) > 0) {
l = m + 1;
} else {
data_len = xdb_le_get_uint16(segment_buffer, d_bytes);
data_ptr = xdb_le_get_uint32(segment_buffer, d_bytes + 2);
break;
}
}
// printf("data_len=%u, data_ptr=%u\n", data_len, data_ptr);
if (data_len == 0) {
err = 100;
goto defer;
}
// buffer alloc checking
err = xdb_region_buffer_alloc(region, data_len);
if (err != 0) {
err += 100;
goto defer;
}
err = read(xdb, data_ptr, region->value, data_len);
if (err != 0) {
err += 30;
goto defer;
}
defer:
// checn and free the segment buffer
if (segment_buffer != NULL) {
xdb_free(segment_buffer);
segment_buffer = NULL;
}
return err;
}
XDB_PRIVATE(int) read(xdb_searcher_t *xdb, long offset, char *buffer, size_t length) {
// check the xdb content cache first
if (xdb->content != NULL) {
memcpy(buffer, xdb->content->buffer + offset, length);
return 0;
}
// seek to the offset
if (fseek(xdb->handle, offset, SEEK_SET) == -1) {
return 1;
}
xdb->io_count++;
if (fread(buffer, 1, length, xdb->handle) != length) {
return 2;
}
return 0;
}
XDB_PUBLIC(xdb_version_t *) xdb_get_version(xdb_searcher_t *xdb) {
return xdb->version;
}
XDB_PUBLIC(int) xdb_get_io_count(xdb_searcher_t *xdb) {
return xdb->io_count;
}

494
binding/c/xdb_util.c
View File

@ -1,494 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/27
#include "xdb_api.h"
#include <ctype.h>
// for Linux
#ifdef XDB_LINUX
#include "sys/time.h"
#elif XDB_WINDOWS
#include <time.h>
#endif
// @Note: since 2023/10/13 to compatible with the windows system
#ifdef XDB_WINDOWS
static int winsock_initialized = 0;
XDB_PUBLIC(int) xdb_init_winsock() {
if (winsock_initialized == 1) {
return 0;
}
WSADATA wsaData;
if (WSAStartup(MAKEWORD(2,2), &wsaData) != 0) {
return -1;
}
winsock_initialized = 1;
return 0;
}
XDB_PUBLIC(void) xdb_clean_winsock() {
if (winsock_initialized == 1) {
WSACleanup();
winsock_initialized = 0;
}
}
XDB_PRIVATE(int) gettimeofday(struct timeval* tp, void* tzp) {
time_t clock;
struct tm tm;
SYSTEMTIME wtm;
GetLocalTime(&wtm);
tm.tm_year = wtm.wYear - 1900;
tm.tm_mon = wtm.wMonth - 1;
tm.tm_mday = wtm.wDay;
tm.tm_hour = wtm.wHour;
tm.tm_min = wtm.wMinute;
tm.tm_sec = wtm.wSecond;
tm.tm_isdst = -1;
clock = mktime(&tm);
tp->tv_sec = clock;
tp->tv_usec = wtm.wMilliseconds * 1000;
return (0);
}
#else
XDB_PUBLIC(int) xdb_init_winsock() {return 0;}
XDB_PUBLIC(void) xdb_clean_winsock() {}
#endif
// --- xdb buffer function implementations
XDB_PUBLIC(xdb_header_t *) xdb_load_header(FILE *handle) {
xdb_header_t *header;
unsigned int size = xdb_header_info_length;
// entry alloc
header = (xdb_header_t *) xdb_malloc(sizeof(xdb_header_t));
if (header == NULL) {
return NULL;
}
if (fseek(handle, 0, SEEK_SET) == -1) {
xdb_free(header);
return NULL;
}
if (fread(header->buffer, 1,size, handle) != size) {
xdb_free(header);
return NULL;
}
// fill the fields
header->length = size;
header->version = (unsigned short) xdb_le_get_uint16(header->buffer, 0);
header->index_policy = (unsigned short) xdb_le_get_uint16(header->buffer, 2);
header->created_at = xdb_le_get_uint32(header->buffer, 4);
header->start_index_ptr = xdb_le_get_uint32(header->buffer, 8);
header->end_index_ptr = xdb_le_get_uint32(header->buffer,12);
// since IPv6 supporting
header->ip_version = xdb_le_get_uint16(header->buffer, 16);
header->runtime_ptr_bytes = xdb_le_get_uint16(header->buffer, 18);
return header;
}
XDB_PUBLIC(xdb_header_t *) xdb_load_header_from_file(const char *db_path) {
xdb_header_t *header;
FILE *handle = fopen(db_path, "rb");
if (handle == NULL) {
return NULL;
}
header = xdb_load_header(handle);
fclose(handle);
return header;
}
XDB_PUBLIC(void) xdb_free_header(void *ptr) {
xdb_header_t *header = (xdb_header_t *) ptr;
if (header->length > 0) {
header->length = 0;
xdb_free(header);
}
}
// --- vector index
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index(FILE *handle) {
xdb_vector_index_t *v_index;
unsigned int size = xdb_vector_index_length;
// seek to the vector index offset
if (fseek(handle, xdb_header_info_length, SEEK_SET) == -1) {
return NULL;
}
// do the buffer read
v_index = (xdb_vector_index_t *) xdb_malloc(sizeof(xdb_vector_index_t));
if (v_index == NULL) {
return NULL;
}
v_index->length = size;
if (fread(v_index->buffer, 1, size, handle) != size) {
xdb_free(v_index);
return NULL;
}
return v_index;
}
XDB_PUBLIC(xdb_vector_index_t *) xdb_load_vector_index_from_file(const char *db_path) {
xdb_vector_index_t *v_index;
FILE *handle = fopen(db_path, "rb");
if (handle == NULL) {
return NULL;
}
v_index = xdb_load_vector_index(handle);
fclose(handle);
return v_index;
}
XDB_PUBLIC(void) xdb_free_vector_index(void *ptr) {
xdb_vector_index_t *v_index = (xdb_vector_index_t *) ptr;
if (v_index->length > 0) {
v_index->length = 0;
xdb_free(v_index);
}
}
// --- content buffer
XDB_PUBLIC(xdb_content_t *) xdb_load_content(FILE *handle) {
unsigned int size;
xdb_content_t *content;
// determine the file size
if (fseek(handle, 0, SEEK_END) == -1) {
return NULL;
}
size = (unsigned int) ftell(handle);
if (fseek(handle, 0, SEEK_SET) == -1) {
return NULL;
}
// do the file read
content = (xdb_content_t *) xdb_malloc(sizeof(xdb_content_t));
if (content == NULL) {
return NULL;
}
// do the buffer alloc
content->buffer = (char *) xdb_malloc(size);
if (content->buffer == NULL) {
xdb_free(content);
return NULL;
}
// read the content into the buffer
content->length = size;
if (fread(content->buffer, 1, size, handle) != size) {
xdb_free(content);
return NULL;
}
return content;
}
XDB_PUBLIC(xdb_content_t *) xdb_load_content_from_file(const char *db_path) {
xdb_content_t *content;
FILE *handle = fopen(db_path, "rb");
if (handle == NULL) {
return NULL;
}
content = xdb_load_content(handle);
fclose(handle);
return content;
}
XDB_PUBLIC(void) xdb_free_content(void *ptr) {
xdb_content_t *content = (xdb_content_t *) ptr;
if (content->length > 0) {
content->length = 0;
xdb_free(content->buffer);
content->buffer = NULL;
xdb_free(content);
}
}
XDB_PUBLIC(int) xdb_verify_from_header(FILE *handle, xdb_header_t *header) {
int runtime_ptr_bytes = 0; // runtime ptr bytes
if (header->version == xdb_structure_20) {
runtime_ptr_bytes = 4;
} else if (header->version == xdb_structure_30) {
runtime_ptr_bytes = header->runtime_ptr_bytes;
} else {
return 2;
}
// 1, confirm the xdb file size.
// to ensure that the maximum file pointer does not overflow.
int err = fseek(handle, 0L, SEEK_END);
if (err != 0) {
return 3;
}
long int fileBytes = ftell(handle);
long int maxFilePtr = (1L << (runtime_ptr_bytes * 8)) - 1;
// printf("fileBytes: %ld, maxFilePtr: %ld\n", fileBytes, maxFilePtr);
if (fileBytes > maxFilePtr) {
return 4;
}
return 0;
}
XDB_PUBLIC(int) xdb_verify(FILE *handle) {
xdb_header_t *header = xdb_load_header(handle);
if (header == NULL) {
return 1;
}
int errcode = xdb_verify_from_header(handle, header);
if (errcode != 0) {
goto done;
}
// what next ?
done:
xdb_free_header(header);
return errcode;
}
XDB_PUBLIC(int) xdb_verify_from_file(const char *db_path) {
FILE *handle = fopen(db_path, "rb");
if (handle == NULL) {
return -1;
}
int r = xdb_verify(handle);
fclose(handle);
return r;
}
// --- End content buffer
// --- ip version
// ip compare for IPv4
// ip1 - with Big endian byte order parsed from an input
// ip2 - with Little endian byte order read from the xdb index.
// to compatiable with the Little Endian encoded IPv4 on xdb 2.0.
XDB_PRIVATE(int) _ipv4_sub_compare(const bytes_ip_t *ip_bytes, int bytes, const char *buffer, int offset) {
register int i0, i1;
for (int i = 0, j = offset + bytes - 1; i < bytes; i++, j--) {
i0 = ip_bytes[i];
i1 = buffer[j] & 0xFF;
if (i0 > i1) {
return 1;
} else if (i0 < i1) {
return -1;
}
}
return 0;
}
static xdb_version_t _ip_version_list[] = {
// 14 = 4 + 4 + 2 + 4
{xdb_ipv4_id, "IPv4", xdb_ipv4_bytes, 14, _ipv4_sub_compare},
// 38 = 16 + 16 + 2 + 4
{xdb_ipv6_id, "IPv6", xdb_ipv6_bytes, 38, xdb_ip_sub_compare},
// END
{0, NULL, 0, 0, NULL}
};
XDB_PUBLIC(xdb_version_t *) xdb_version_v4() {
return &_ip_version_list[0];
}
XDB_PUBLIC(xdb_version_t *) xdb_version_v6() {
return &_ip_version_list[1];
}
XDB_PUBLIC(int) xdb_version_is_v4(const xdb_version_t *version) {
return version->id == xdb_ipv4_id;
}
XDB_PUBLIC(int) xdb_version_is_v6(const xdb_version_t *version) {
return version->id == xdb_ipv6_id;
}
XDB_PUBLIC(xdb_version_t *) xdb_version_from_name(char *name) {
// to upper case the name
for (int i = 0; name[i] != '\0'; i++) {
name[i] = toupper((unsigned char) name[i]);
}
if (strcmp(name, "V4") == 0 || strcmp(name, "IPV4") == 0) {
return xdb_version_v4();
} else if (strcmp(name, "V6") == 0 || strcmp(name, "IPV6") == 0) {
return xdb_version_v6();
} else {
return NULL;
}
}
XDB_PUBLIC(xdb_version_t *) xdb_version_from_header(xdb_header_t *header) {
// Old structure with ONLY IPv4 supports
if (header->version == xdb_structure_20) {
return xdb_version_v4();
}
// structure 3.0 with IPv6 supporting
if (header->version != xdb_structure_30) {
return NULL;
}
if (header->ip_version == xdb_ipv4_id) {
return xdb_version_v4();
} else if (header->ip_version == xdb_ipv6_id) {
return xdb_version_v6();
} else {
return NULL;
}
}
// --- END ip version
XDB_PUBLIC(long) xdb_now() {
struct timeval c_time;
gettimeofday(&c_time, NULL);
return c_time.tv_sec * (int)1e6 + c_time.tv_usec;
}
XDB_PUBLIC(unsigned int) xdb_le_get_uint32(const char *buffer, int offset) {
return (
((buffer[offset ]) & 0x000000FF) |
((buffer[offset+1] << 8) & 0x0000FF00) |
((buffer[offset+2] << 16) & 0x00FF0000) |
((buffer[offset+3] << 24) & 0xFF000000)
);
}
XDB_PUBLIC(int) xdb_le_get_uint16(const char *buffer, int offset) {
return (
((buffer[offset ]) & 0x000000FF) |
((buffer[offset+1] << 8) & 0x0000FF00)
);
}
XDB_PUBLIC(xdb_version_t *) xdb_parse_ip(const string_ip_t *ip_string, bytes_ip_t *buffer, size_t length) {
// version check
if (strchr(ip_string, '.') != NULL && strchr(ip_string, ':') == NULL) {
return xdb_parse_v4_ip(ip_string, buffer, length);
} else if (strchr(ip_string, ':') != NULL) {
return xdb_parse_v6_ip(ip_string, buffer, length);
}
return NULL;
}
XDB_PUBLIC(xdb_version_t *) xdb_parse_v4_ip(const string_ip_t *ip_string, bytes_ip_t *buffer, size_t length) {
struct in_addr addr;
// buffer length checking
if (length < xdb_ipv4_bytes) {
return NULL;
}
if (inet_pton(AF_INET, ip_string, &addr) != 1) {
return NULL;
}
// encode the address to buffer with big endian byte bufffer.
buffer[0] = (addr.s_addr) & 0xFF;
buffer[1] = (addr.s_addr >> 8) & 0xFF;
buffer[2] = (addr.s_addr >> 16) & 0xFF;
buffer[3] = (addr.s_addr >> 24) & 0xFF;
return XDB_IPv4;
}
XDB_PUBLIC(xdb_version_t *) xdb_parse_v6_ip(const string_ip_t *ip_string, bytes_ip_t *buffer, size_t length) {
struct in6_addr addr;
// buffer length checking
if (length < xdb_ipv6_bytes) {
return NULL;
}
if (inet_pton(AF_INET6, ip_string, &addr) != 1) {
return NULL;
}
memcpy(buffer, addr.s6_addr, xdb_ipv6_bytes);
return XDB_IPv6;
}
XDB_PUBLIC(int) xdb_ip_to_string(const bytes_ip_t *ip_bytes, int bytes, char *ip_string, size_t length) {
if (bytes == xdb_ipv4_bytes) {
return xdb_v4_ip_to_string(ip_bytes, ip_string, length);
} else if (bytes == xdb_ipv6_bytes) {
return xdb_v6_ip_to_string(ip_bytes, ip_string, length);
}
return -1;
}
XDB_PUBLIC(int) xdb_v4_ip_to_string(const bytes_ip_t *ip_bytes, char *ip_string, size_t length) {
if (!ip_bytes || !ip_string || length == 0) {
return -1;
}
// buffer length checking
if (length < INET_ADDRSTRLEN) {
return -1;
}
if (inet_ntop(AF_INET, ip_bytes, ip_string, length) == NULL) {
return -1;
}
return 0;
}
XDB_PUBLIC(int) xdb_v6_ip_to_string(const bytes_ip_t *ip_bytes, char *ip_string, size_t length) {
if (!ip_bytes || !ip_string || length == 0) {
return -1;
}
if (length < INET6_ADDRSTRLEN) {
return -1;
}
if (inet_ntop(AF_INET6, ip_bytes, ip_string, length) == NULL) {
return -1;
}
return 0;
}
XDB_PUBLIC(int) xdb_ip_sub_compare(const bytes_ip_t *ip1, int bytes, const char *buffer, int offset) {
register int i, i1, i2;
for (i = 0; i < bytes; i++) {
i1 = ip1[i];
i2 = buffer[offset + i] & 0xFF;
if (i1 > i2) {
return 1;
} else if (i1 < i2) {
return -1;
}
}
return 0;
}

11
binding/cpp/Makefile
View File

@ -1,11 +0,0 @@
all: xdb_search xdb_bench
xdb_search: xdb_search.cc xdb_search_test.cc
g++ -std=c++11 -O2 $^ -o $@
xdb_bench: xdb_search.cc xdb_bench.cc xdb_bench_test.cc
g++ -std=c++11 -O2 $^ -o $@
clean:
rm -f xdb_search xdb_bench

108
binding/cpp/readme.md
View File

@ -1,108 +0,0 @@
# ip2region xdb C++ 查询客户端实现
## 使用方式
### 完全基于文件的查询
```
#include <iostream>
#include "xdb_search.h"
int main(int argc, char* argv[]) {
char file_name[] = "../../data/ip2region.xdb";
char ip[] = "1.2.3.4";
xdb_search_t xdb(file_name);
xdb.init_file();
std::cout << xdb.search(ip) << std::endl;
return 0;
}
```
### 缓存 `vector_index` 索引
```
#include <iostream>
#include "xdb_search.h"
int main(int argc, char* argv[]) {
char file_name[] = "../../data/ip2region.xdb";
char ip[] = "1.2.3.4";
xdb_search_t xdb(file_name);
xdb.init_vector_index();
std::cout << xdb.search(ip) << std::endl;
return 0;
}
```
### 缓存整个 `xdb` 数据
```
#include <iostream>
#include "xdb_search.h"
int main(int argc, char* argv[]) {
char file_name[] = "../../data/ip2region.xdb";
char ip[] = "1.2.3.4";
xdb_search_t xdb(file_name);
xdb.init_content();
std::cout << xdb.search(ip) << std::endl;
return 0;
}
```
## 测试程序编译
1. 切换到当前目录
2. 编译
```
$ make
g++ -std=c++11 -O2 xdb_search.cc xdb_search_test.cc -o xdb_search
g++ -std=c++11 -O2 xdb_search.cc xdb_bench.cc xdb_bench_test.cc -o xdb_bench
```
## 测试查询
### 说明
```
$ ./xdb_search --help
./xdb_search [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vector_index/content
--help print help
```
### 测试
```
$ ./xdb_search --db ../../data/ip2region.xdb --cache-policy vector_index
cache policy : vector_index
ip2region>> 1.2.3.4
美国|0|华盛顿|0|谷歌
```
## bench 测试
### 说明
```
$ ./xdb_bench --help
./xdb_bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vector_index/content
--help print help
```
### 测试
```
$ ./xdb_bench --db ../../data/ip2region.xdb --src ../../data/ip.merge.txt --cache-policy content
total: 3419220, took: 3.44 s, cost: 0.27 μs/op, io count: 0
$ ./xdb_bench --db ../../data/ip2region.xdb --src ../../data/ip.merge.txt --cache-policy vector_index
total: 3419220, took: 45.99 s, cost: 12.24 μs/op, io count: 21739300
$ ./xdb_bench --db ../../data/ip2region.xdb --src ../../data/ip.merge.txt --cache-policy file
total: 3419220, took: 60.39 s, cost: 16.32 μs/op, io count: 25158520
```

135
binding/cpp/xdb_bench.cc
View File

@ -1,135 +0,0 @@
#include "xdb_bench.h"
#include <arpa/inet.h>
#include <stdio.h>
#include <string.h>
#include <sys/time.h>
#include <iostream>
#include <vector>
static void log_exit(const std::string &msg) {
std::cout << msg << std::endl;
exit(-1);
}
static unsigned long long get_time() {
struct timeval tv1;
gettimeofday(&tv1, NULL);
return (unsigned long long)tv1.tv_sec * 1000 * 1000 + tv1.tv_usec;
}
static bool ip2uint(const char *buf, unsigned int &ip) {
struct in_addr addr;
if (inet_pton(AF_INET, buf, &addr) == 0)
return false;
// 网络字节序为大端存储, 在此转换为小端存储
ip = (((addr.s_addr >> 0) & 0xFF) << 24) |
(((addr.s_addr >> 8) & 0xFF) << 16) |
(((addr.s_addr >> 16) & 0xFF) << 8) |
(((addr.s_addr >> 24) & 0xFF) << 0);
return true;
}
static std::string uint2ip(unsigned int ip) {
char buf[16];
snprintf(buf,
sizeof(buf),
"%d.%d.%d.%d",
(ip >> 24) & 0xFF,
(ip >> 16) & 0xFF,
(ip >> 8) & 0xFF,
ip & 0xFF);
return std::string(buf);
}
xdb_bench_t::xdb_bench_t(const std::string &file_name) : xdb_search(file_name) {
}
void xdb_bench_t::init_file() {
xdb_search.init_file();
}
void xdb_bench_t::init_vector_index() {
xdb_search.init_vector_index();
}
void xdb_bench_t::init_content() {
xdb_search.init_content();
}
void xdb_bench_t::bench_test_one(unsigned int ip_uint, const char *region) {
if (xdb_search.search(uint2ip(ip_uint)) != region)
log_exit("failed: " + uint2ip(ip_uint));
sum_io_count += xdb_search.get_io_count();
sum_cost_time += xdb_search.get_cost_time();
sum_count++;
}
void xdb_bench_t::bench_test_line(char *buf) {
size_t buf_len = strlen(buf);
if (buf_len == 0)
return;
buf[buf_len - 1] = '\0'; // 去掉换行符
char *pos1 = strchr(buf, '|');
if (pos1 == NULL)
log_exit("invalid data: " + std::string(buf));
char *pos2 = strchr(pos1 + 1, '|');
if (pos2 == NULL)
log_exit("invalid data: " + std::string(buf));
*pos1 = '\0';
*pos2 = '\0';
unsigned int ip1, ip2;
if (!ip2uint(buf, ip1) || !ip2uint(pos1 + 1, ip2) || ip1 > ip2) {
*pos1 = *pos2 = '|';
log_exit(std::string("invalid data: ") + buf);
}
const char *region = pos2 + 1;
unsigned int ip_mid = ip1 + (ip2 - ip1) / 2;
std::vector<unsigned int> ip_vec;
ip_vec.push_back(ip1);
ip_vec.push_back(ip1 + (ip_mid - ip1) / 2);
ip_vec.push_back(ip_mid);
ip_vec.push_back(ip_mid + (ip2 - ip_mid) / 2);
ip_vec.push_back(ip2);
for (auto &d : ip_vec)
bench_test_one(d, region);
}
void xdb_bench_t::bench_test_file(const std::string &file_name) {
FILE *f = fopen(file_name.data(), "r");
if (f == NULL)
log_exit("can't open " + file_name);
char buf[1024];
while (fgets(buf, sizeof(buf), f) != NULL)
bench_test_line(buf);
}
void xdb_bench_t::bench(const std::string &file_name) {
sum_io_count = 0;
sum_cost_time = 0;
sum_count = 0;
unsigned long long tv1 = get_time();
bench_test_file(file_name);
unsigned long long tv2 = get_time();
double took = (tv2 - tv1) * 1.0 / 1000 / 1000;
double cost = sum_cost_time * 1.0 / sum_count;
printf(
"total: %llu, took: %.2f s, cost: %.2f μs/op, io "
"count: "
"%llu\n",
sum_count,
took,
cost,
sum_io_count);
}

28
binding/cpp/xdb_bench.h
View File

@ -1,28 +0,0 @@
#ifndef XDB_BENCH_H
#define XDB_BENCH_H
#include "xdb_search.h"
class xdb_bench_t {
public:
xdb_bench_t(const std::string &file_name);
void init_file();
void init_vector_index();
void init_content();
void bench(const std::string &file_name);
private:
void bench_test_one(unsigned int ip_uint, const char *region);
void bench_test_line(char *buf);
void bench_test_file(const std::string &file_name);
xdb_search_t xdb_search;
unsigned long long sum_io_count;
unsigned long long sum_cost_time;
unsigned long long sum_count;
};
#endif

70
binding/cpp/xdb_bench_test.cc
View File

@ -1,70 +0,0 @@
#include "xdb_bench.h"
#include <getopt.h>
#include <iostream>
void print_help(int argc, char* argv[]) {
printf("./xdb_bench [command options]\n");
printf("options:\n");
printf(" --db string ip2region binary xdb file path\n");
printf(" --src string source ip text file path\n");
printf(
" --cache-policy string cache policy: "
"file/vector_index/content\n");
printf(" --help print help\n");
exit(-1);
}
int main(int argc, char* argv[]) {
struct option long_options[] = {
{"db", required_argument, 0, 'd'},
{"cache-policy", required_argument, 0, 't'},
{"src", required_argument, 0, 's'},
{"help", no_argument, 0, 'h'},
{0, 0, 0, 0 }
};
std::string db_file_name = "../../data/ip2region.xdb";
std::string src_file_name = "../../data/ip.merge.txt";
std::string cache_policy = "vector_index";
while (1) {
int c = getopt_long(argc, argv, "", long_options, NULL);
if (c == -1)
break;
switch (c) {
case 'd':
db_file_name = optarg;
break;
case 'h':
print_help(argc, argv);
break;
case 't':
cache_policy = optarg;
break;
case 's':
src_file_name = optarg;
break;
case '?':
exit(-1);
}
}
xdb_bench_t xdb(db_file_name);
if (cache_policy == "content")
xdb.init_content();
else if (cache_policy == "vector_index")
xdb.init_vector_index();
else if (cache_policy == "file")
xdb.init_file();
else {
std::cout << "invalid cache policy: " << cache_policy << std::endl;
exit(-1);
}
xdb.bench(src_file_name);
return 0;
}

190
binding/cpp/xdb_search.cc
View File

@ -1,190 +0,0 @@
#include "xdb_search.h"
#include <arpa/inet.h>
#include <sys/time.h>
#include <iostream>
static void log_exit(const std::string &msg) {
std::cout << msg << std::endl;
exit(-1);
}
static unsigned long long get_time() {
struct timeval tv1;
gettimeofday(&tv1, NULL);
return (unsigned long long)tv1.tv_sec * 1000 * 1000 + tv1.tv_usec;
}
static void read_bin(int index, char *buf, size_t len, FILE *db) {
fseek(db, index, SEEK_SET);
if (fread(buf, 1, len, db) != len)
log_exit(__func__);
}
static unsigned int read_uint(const char *buf) {
return ((buf[0]) & 0x000000FF) | ((buf[1] << 8) & 0x0000FF00) |
((buf[2] << 16) & 0x00FF0000) | ((buf[3] << 24) & 0xFF000000);
}
static unsigned short read_ushort(const char *buf) {
return ((buf[0]) & 0x000000FF) | ((buf[1] << 8) & 0x0000FF00);
}
static bool ip2uint(const char *buf, unsigned int &ip) {
struct in_addr addr;
if (inet_pton(AF_INET, buf, &addr) == 0)
return false;
// 网络字节序为大端存储, 在此转换为小端存储
ip = (((addr.s_addr >> 0) & 0xFF) << 24) |
(((addr.s_addr >> 8) & 0xFF) << 16) |
(((addr.s_addr >> 16) & 0xFF) << 8) |
(((addr.s_addr >> 24) & 0xFF) << 0);
return true;
}
void xdb_search_t::get_content_index(unsigned int ip,
unsigned int &left,
unsigned int &right) {
unsigned int ip_1 = (ip >> 24) & 0xFF;
unsigned int ip_2 = (ip >> 16) & 0xFF;
unsigned int index = (ip_1 * vector_index_cols + ip_2) * vector_index_size;
if (content != NULL) {
left = read_uint(content + index + header_length);
right = read_uint(content + index + header_length + 4);
} else if (vector_index != NULL) {
left = read_uint(vector_index + index);
right = read_uint(vector_index + index + 4);
} else {
++io_count;
char buf[8];
read_bin(header_length + index, buf, sizeof(buf), db);
left = read_uint(buf);
right = read_uint(buf + 4);
}
}
void xdb_search_t::get_content(unsigned int index,
unsigned int &ip_left,
unsigned int &ip_right,
unsigned short &region_len,
unsigned int &region_index) {
char buf[segment_index_size]; // 4 + 4 + 2 + 4
const char *p;
if (content != NULL) {
p = content + index;
} else {
++io_count;
read_bin(index, buf, sizeof(buf), db);
p = buf;
}
ip_left = read_uint(p);
ip_right = read_uint(p + 4);
region_len = read_ushort(p + 8);
region_index = read_uint(p + 10);
}
std::string xdb_search_t::get_region(unsigned int index, unsigned short len) {
if (content != NULL) {
return std::string(content + index, len);
} else {
++io_count;
char *buf = (char *)malloc(sizeof(char) * len);
read_bin(index, buf, len, db);
std::string res(buf, len);
free(buf);
return res;
}
}
xdb_search_t::xdb_search_t(const std::string &file_name) {
db = fopen(file_name.data(), "r");
vector_index = NULL;
content = NULL;
if (db == NULL)
log_exit("can't open " + file_name);
}
void xdb_search_t::init_file() {
}
void xdb_search_t::init_vector_index() {
vector_index = (char *)malloc(vector_index_length);
read_bin(header_length, vector_index, vector_index_length, db);
}
void xdb_search_t::init_content() {
fseek(db, 0, SEEK_END);
unsigned int size = ftell(db);
content = (char *)malloc(size);
read_bin(0, content, size, db);
}
xdb_search_t::~xdb_search_t() {
if (db != NULL) {
fclose(db);
db = NULL;
}
if (vector_index != NULL) {
free(vector_index);
vector_index = NULL;
}
if (content != NULL) {
free(content);
content = NULL;
}
}
unsigned long long xdb_search_t::get_io_count() {
return io_count;
}
unsigned long long xdb_search_t::get_cost_time() {
return cost_time;
}
std::string xdb_search_t::search(const std::string &ip_str) {
unsigned long long t1 = get_time();
unsigned int ip_uint;
if (!ip2uint(ip_str.data(), ip_uint))
return "invalid ip: " + ip_str;
std::string region = search(ip_uint);
unsigned long long t2 = get_time();
cost_time = t2 - t1;
return region;
}
std::string xdb_search_t::search(unsigned int ip_uint) {
io_count = 0;
unsigned int content_index_left, content_index_right;
get_content_index(ip_uint, content_index_left, content_index_right);
unsigned int left, right, mid;
unsigned int ip_left, ip_right;
unsigned short region_len;
unsigned int region_index;
unsigned int mid_index;
left = 0;
right = (content_index_right - content_index_left) / segment_index_size;
for (;;) {
mid = left + (right - left) / 2;
mid_index = content_index_left + mid * segment_index_size;
get_content(mid_index, ip_left, ip_right, region_len, region_index);
if (ip_left > ip_uint)
right = mid - 1;
else if (ip_right < ip_uint)
left = mid + 1;
else
return get_region(region_index, region_len);
}
}

50
binding/cpp/xdb_search.h
View File

@ -1,50 +0,0 @@
#ifndef XDB_SEARCH_H
#define XDB_SEARCH_H
#include <string>
class xdb_search_t {
public:
xdb_search_t(const std::string &file_name);
~xdb_search_t();
void init_file();
void init_vector_index();
void init_content();
unsigned long long get_io_count();
unsigned long long get_cost_time();
std::string search(const std::string &ip);
private:
void get_content_index(unsigned int ip,
unsigned int &left,
unsigned int &right);
void get_content(unsigned int index,
unsigned int &ip_left,
unsigned int &ip_right,
unsigned short &region_len,
unsigned int &region_index);
std::string get_region(unsigned int index, unsigned short len);
std::string search(unsigned int ip_uint);
FILE *db;
char *vector_index;
char *content;
unsigned long long io_count;
unsigned long long cost_time;
static constexpr int header_length = 256;
static constexpr int vector_index_rows = 256;
static constexpr int vector_index_cols = 256;
static constexpr int vector_index_size = 8;
static constexpr int vector_index_length =
vector_index_rows * vector_index_cols * vector_index_size;
static constexpr int segment_index_size = 14;
};
#endif

73
binding/cpp/xdb_search_test.cc
View File

@ -1,73 +0,0 @@
#include "xdb_search.h"
#include <getopt.h>
#include <iostream>
void print_help(int argc, char* argv[]) {
printf("./xdb_search [command options]\n");
printf("options:\n");
printf(" --db string ip2region binary xdb file path\n");
printf(
" --cache-policy string cache policy: "
"file/vector_index/content\n");
printf(" --help print help\n");
exit(-1);
}
int main(int argc, char* argv[]) {
struct option long_options[] = {
{"db", required_argument, 0, 'd'},
{"cache-policy", required_argument, 0, 't'},
{"help", no_argument, 0, 'h'},
{0, 0, 0, 0 }
};
std::string db_file_name = "../../data/ip2region.xdb";
std::string cache_policy = "vector_index";
while (1) {
int c = getopt_long(argc, argv, "", long_options, NULL);
if (c == -1)
break;
switch (c) {
case 'd':
db_file_name = optarg;
break;
case 'h':
print_help(argc, argv);
break;
case 't':
cache_policy = optarg;
break;
case '?':
exit(-1);
}
}
xdb_search_t xdb(db_file_name);
if (cache_policy == "content")
xdb.init_content();
else if (cache_policy == "vector_index")
xdb.init_vector_index();
else if (cache_policy == "file")
xdb.init_file();
else {
std::cout << "invalid cache policy: " << cache_policy << std::endl;
exit(-1);
}
std::string ip;
for (;;) {
std::cout << "ip2region>> ";
std::getline(std::cin, ip);
if (ip.empty())
continue;
if (ip == "exit" || ip == "quit")
break;
std::cout << xdb.search(ip) << std::endl;
}
return 0;
}

337
binding/csharp/.gitignore vendored
View File

@ -1,337 +0,0 @@
## Ignore Visual Studio temporary files, build results, and
## files generated by popular Visual Studio add-ons.
##
## Get latest from https://github.com/github/gitignore/blob/master/VisualStudio.gitignore
# User-specific files
*.rsuser
*.suo
*.user
*.userosscache
*.sln.docstates
# User-specific files (MonoDevelop/Xamarin Studio)
*.userprefs
# Build results
[Dd]ebug/
[Dd]ebugPublic/
[Rr]elease/
[Rr]eleases/
x64/
x86/
[Aa][Rr][Mm]/
[Aa][Rr][Mm]64/
bld/
[Bb]in/
[Oo]bj/
[Ll]og/
# Visual Studio 2015/2017 cache/options directory
.vs/
# Uncomment if you have tasks that create the project's static files in wwwroot
#wwwroot/
# Visual Studio 2017 auto generated files
Generated\ Files/
# MSTest test Results
[Tt]est[Rr]esult*/
[Bb]uild[Ll]og.*
# NUNIT
*.VisualState.xml
TestResult.xml
# Build Results of an ATL Project
[Dd]ebugPS/
[Rr]eleasePS/
dlldata.c
# Benchmark Results
BenchmarkDotNet.Artifacts/
# .NET Core
project.lock.json
project.fragment.lock.json
artifacts/
# StyleCop
StyleCopReport.xml
# Files built by Visual Studio
*_i.c
*_p.c
*_h.h
*.ilk
*.meta
*.obj
*.iobj
*.pch
*.pdb
*.ipdb
*.pgc
*.pgd
*.rsp
*.sbr
*.tlb
*.tli
*.tlh
*.tmp
*.tmp_proj
*_wpftmp.csproj
*.log
*.vspscc
*.vssscc
.builds
*.pidb
*.svclog
*.scc
# Chutzpah Test files
_Chutzpah*
# Visual C++ cache files
ipch/
*.aps
*.ncb
*.opendb
*.opensdf
*.sdf
*.cachefile
*.VC.db
*.VC.VC.opendb
# Visual Studio profiler
*.psess
*.vsp
*.vspx
*.sap
# Visual Studio Trace Files
*.e2e
# TFS 2012 Local Workspace
$tf/
# Guidance Automation Toolkit
*.gpState
# ReSharper is a .NET coding add-in
_ReSharper*/
*.[Rr]e[Ss]harper
*.DotSettings.user
# JustCode is a .NET coding add-in
.JustCode
# TeamCity is a build add-in
_TeamCity*
# DotCover is a Code Coverage Tool
*.dotCover
# AxoCover is a Code Coverage Tool
.axoCover/*
!.axoCover/settings.json
# Visual Studio code coverage results
*.coverage
*.coveragexml
# NCrunch
_NCrunch_*
.*crunch*.local.xml
nCrunchTemp_*
# MightyMoose
*.mm.*
AutoTest.Net/
# Web workbench (sass)
.sass-cache/
# Installshield output folder
[Ee]xpress/
# DocProject is a documentation generator add-in
DocProject/buildhelp/
DocProject/Help/*.HxT
DocProject/Help/*.HxC
DocProject/Help/*.hhc
DocProject/Help/*.hhk
DocProject/Help/*.hhp
DocProject/Help/Html2
DocProject/Help/html
# Click-Once directory
publish/
# Publish Web Output
*.[Pp]ublish.xml
*.azurePubxml
# Note: Comment the next line if you want to checkin your web deploy settings,
# but database connection strings (with potential passwords) will be unencrypted
*.pubxml
*.publishproj
# Microsoft Azure Web App publish settings. Comment the next line if you want to
# checkin your Azure Web App publish settings, but sensitive information contained
# in these scripts will be unencrypted
PublishScripts/
# NuGet Packages
*.nupkg
# The packages folder can be ignored because of Package Restore
**/[Pp]ackages/*
# except build/, which is used as an MSBuild target.
!**/[Pp]ackages/build/
# Uncomment if necessary however generally it will be regenerated when needed
#!**/[Pp]ackages/repositories.config
# NuGet v3's project.json files produces more ignorable files
*.nuget.props
*.nuget.targets
# Microsoft Azure Build Output
csx/
*.build.csdef
# Microsoft Azure Emulator
ecf/
rcf/
# Windows Store app package directories and files
AppPackages/
BundleArtifacts/
Package.StoreAssociation.xml
_pkginfo.txt
*.appx
# Visual Studio cache files
# files ending in .cache can be ignored
*.[Cc]ache
# but keep track of directories ending in .cache
!?*.[Cc]ache/
# Others
ClientBin/
~$*
*~
*.dbmdl
*.dbproj.schemaview
*.jfm
*.pfx
*.publishsettings
orleans.codegen.cs
# Including strong name files can present a security risk
# (https://github.com/github/gitignore/pull/2483#issue-259490424)
#*.snk
# Since there are multiple workflows, uncomment next line to ignore bower_components
# (https://github.com/github/gitignore/pull/1529#issuecomment-104372622)
#bower_components/
# RIA/Silverlight projects
Generated_Code/
# Backup & report files from converting an old project file
# to a newer Visual Studio version. Backup files are not needed,
# because we have git ;-)
_UpgradeReport_Files/
Backup*/
UpgradeLog*.XML
UpgradeLog*.htm
ServiceFabricBackup/
*.rptproj.bak
# SQL Server files
*.mdf
*.ldf
*.ndf
# Business Intelligence projects
*.rdl.data
*.bim.layout
*.bim_*.settings
*.rptproj.rsuser
*- Backup*.rdl
# Microsoft Fakes
FakesAssemblies/
# GhostDoc plugin setting file
*.GhostDoc.xml
# Node.js Tools for Visual Studio
.ntvs_analysis.dat
node_modules/
# Visual Studio 6 build log
*.plg
# Visual Studio 6 workspace options file
*.opt
# Visual Studio 6 auto-generated workspace file (contains which files were open etc.)
*.vbw
# Visual Studio LightSwitch build output
**/*.HTMLClient/GeneratedArtifacts
**/*.DesktopClient/GeneratedArtifacts
**/*.DesktopClient/ModelManifest.xml
**/*.Server/GeneratedArtifacts
**/*.Server/ModelManifest.xml
_Pvt_Extensions
# Paket dependency manager
.paket/paket.exe
paket-files/
# FAKE - F# Make
.fake/
# JetBrains Rider
.idea/
*.sln.iml
# CodeRush personal settings
.cr/personal
# Python Tools for Visual Studio (PTVS)
__pycache__/
*.pyc
# Cake - Uncomment if you are using it
# tools/**
# !tools/packages.config
# Tabs Studio
*.tss
# Telerik's JustMock configuration file
*.jmconfig
# BizTalk build output
*.btp.cs
*.btm.cs
*.odx.cs
*.xsd.cs
# OpenCover UI analysis results
OpenCover/
# Azure Stream Analytics local run output
ASALocalRun/
# MSBuild Binary and Structured Log
*.binlog
# NVidia Nsight GPU debugger configuration file
*.nvuser
# MFractors (Xamarin productivity tool) working folder
.mfractor/
# Local History for Visual Studio
.localhistory/

21
binding/csharp/CHANGELOG.md
View File

@ -1,21 +0,0 @@
# Changelog
All notable changes to this project will be documented in this file.
## [Unreleased]
## [2.0.1] - 2023-07-30
### Added
- Support netstandard2.0
## [2.0.0] - 2023-07-26
### Removed
- Remove nuget include xdb file
- Searcher cache policy default parameters
- Searcher xdb file path default parameters
### Added
- Dependent file query policies CachePolicy.VectorIndex, CachePolicy.File support thread-safe concurrent queries
- Dramatically optimizes overall performance

26
binding/csharp/IP2Region.Net.BenchMark/IP2Region.Net.BenchMark.csproj
View File

@ -1,26 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net6.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.13.2" />
<PackageReference Include="BenchmarkDotNet.Annotations" Version="0.13.2" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\IP2Region.Net\IP2Region.Net.csproj" />
</ItemGroup>
<ItemGroup>
<Content Include="..\..\..\data\ip2region.xdb">
<Link>IP2Region/ip2region.xdb</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
</Project>

29
binding/csharp/IP2Region.Net.BenchMark/Program.cs
View File

@ -1,29 +0,0 @@
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
using IP2Region.Net.Abstractions;
using IP2Region.Net.XDB;
BenchmarkRunner.Run(typeof(Program).Assembly);
public class CachePolicyCompare
{
private static readonly string XdbPath = Path.Combine(AppContext.BaseDirectory, "IP2Region", "ip2region.xdb");
private readonly ISearcher _contentSearcher = new Searcher(CachePolicy.Content, XdbPath);
private readonly ISearcher _vectorSearcher = new Searcher(CachePolicy.VectorIndex,XdbPath);
private readonly ISearcher _fileSearcher = new Searcher(CachePolicy.File,XdbPath);
private readonly string _testIpAddress = "114.114.114.114";
[Benchmark]
[BenchmarkCategory(nameof(CachePolicy.Content))]
public void CachePolicy_Content() => _contentSearcher.Search(_testIpAddress);
[Benchmark]
[BenchmarkCategory(nameof(CachePolicy.VectorIndex))]
public void CachePolicy_VectorIndex() => _vectorSearcher.Search(_testIpAddress);
[Benchmark]
[BenchmarkCategory(nameof(CachePolicy.File))]
public void CachePolicy_File() => _fileSearcher.Search(_testIpAddress);
}

34
binding/csharp/IP2Region.Net.Test/IP2Region.Net.Test.csproj
View File

@ -1,34 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<TargetFramework>net6.0</TargetFramework>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<IsPackable>false</IsPackable>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.1.0" />
<PackageReference Include="NUnit" Version="3.13.3" />
<PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
<PackageReference Include="NUnit.Analyzers" Version="3.3.0" />
<PackageReference Include="coverlet.collector" Version="3.1.2" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\IP2Region.Net\IP2Region.Net.csproj" />
</ItemGroup>
<ItemGroup>
<Content Include="..\..\..\data\ip.merge.txt">
<Link>TestData/ip.merge.txt</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
<Content Include="..\..\..\data\ip2region.xdb">
<Link>TestData/ip2region.xdb</Link>
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</Content>
</ItemGroup>
</Project>

80
binding/csharp/IP2Region.Net.Test/SearcherTest.cs
View File

@ -1,80 +0,0 @@
using IP2Region.Net.XDB;
namespace IP2Region.Net.Test;
[TestFixture]
public class SearcherTest
{
private readonly string _xdbPath = Path.Combine(AppContext.BaseDirectory, "TestData", "ip2region.xdb");
public static IEnumerable<string> Ips()
{
yield return "114.114.114.114";
yield return "119.29.29.29";
yield return "223.5.5.5";
yield return "180.76.76.76";
yield return "8.8.8.8";
}
[TestCaseSource(nameof(Ips))]
[Parallelizable(ParallelScope.All)]
public void TestSearchCacheContent(string ip)
{
var contentSearcher = new Searcher(CachePolicy.Content,_xdbPath);
var region = contentSearcher.Search(ip);
Console.WriteLine(region);
}
[TestCaseSource(nameof(Ips))]
[Parallelizable(ParallelScope.All)]
public void TestSearchCacheVector(string ip)
{
var vectorSearcher = new Searcher(CachePolicy.VectorIndex,_xdbPath);
var region = vectorSearcher.Search(ip);
Console.WriteLine(region);
}
[TestCaseSource(nameof(Ips))]
[Parallelizable(ParallelScope.All)]
public void TestSearchCacheFile(string ip)
{
var fileSearcher = new Searcher(CachePolicy.File,_xdbPath);
var region = fileSearcher.Search(ip);
Console.WriteLine(region);
}
[TestCase(CachePolicy.Content)]
[TestCase(CachePolicy.VectorIndex)]
[TestCase(CachePolicy.File)]
public void TestBenchSearch(CachePolicy cachePolicy)
{
Searcher searcher = new Searcher(cachePolicy,_xdbPath);
var srcPath = Path.Combine(AppContext.BaseDirectory, "TestData", "ip.merge.txt");
foreach (var line in File.ReadLines(srcPath))
{
var ps = line.Trim().Split("|", 3);
if (ps.Length != 3)
{
throw new ArgumentException($"invalid ip segment line {line}", nameof(line));
}
var sip = Util.IpAddressToUInt32(ps[0]);
var eip = Util.IpAddressToUInt32(ps[1]);
var mip = Util.GetMidIp(sip, eip);
uint[] temp = { sip, Util.GetMidIp(sip, mip), mip, Util.GetMidIp(mip, eip), eip };
foreach (var ip in temp)
{
var region = searcher.Search(ip);
if (region != ps[2])
{
throw new Exception($"failed search {ip} with ({region}!={ps[2]})");
}
}
}
}
}

1
binding/csharp/IP2Region.Net.Test/Usings.cs
View File

@ -1 +0,0 @@
global using NUnit.Framework;

14
binding/csharp/IP2Region.Net.Test/UtilTest.cs
View File

@ -1,14 +0,0 @@
using IP2Region.Net.XDB;
namespace IP2Region.Net.Test;
[TestFixture]
public class UtilTest
{
[TestCase("114.114.114.114")]
public void TestIpAddressToUInt32(string value)
{
var uintIp = XDB.Util.IpAddressToUInt32(value);
Console.WriteLine(uintIp);
}
}

20
binding/csharp/IP2Region.Net/Abstractions/ISearcher.cs
View File

@ -1,20 +0,0 @@
// Copyright 2023 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Alan <lzh.shap@gmail.com>
// @Date 2023/07/25
using System.Net;
namespace IP2Region.Net.Abstractions;
public interface ISearcher
{
string? Search(string ipStr);
string? Search(IPAddress ipAddress);
string? Search(uint ipAddress);
int IoCount { get; }
}

34
binding/csharp/IP2Region.Net/IP2Region.Net.csproj
View File

@ -1,34 +0,0 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<id>IP2Region.Net</id>
<version>2.0.2</version>
<title>IP2Region.Net</title>
<authors>Alan Lee</authors>
<PackageLicenseExpression>Apache-2.0</PackageLicenseExpression>
<PackageReadmeFile>README.md</PackageReadmeFile>
<PackageProjectUrl>https://github.com/lionsoul2014/ip2region/tree/master/binding/csharp</PackageProjectUrl>
<RepositoryUrl>https://github.com/lionsoul2014/ip2region/tree/master/binding/csharp</RepositoryUrl>
<PackageReleaseNotes>Please refer to CHANGELOG.md for details</PackageReleaseNotes>
<Description>.NET client library for ip2region</Description>
<PackageTags>IP2Region GeoIP IPSearch</PackageTags>
<RepositoryType>git</RepositoryType>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<TargetFrameworks>netstandard2.0;netstandard2.1;net6.0;net7.0</TargetFrameworks>
<LangVersion>10.0</LangVersion>
<UserSecretsId>c2f07fe1-a300-4de3-8200-1278ed8cb5b7</UserSecretsId>
</PropertyGroup>
<ItemGroup>
<None Include="..\README.md" Pack="true" PackagePath="\" />
</ItemGroup>
<ItemGroup Condition="'$(TargetFramework)' == 'netstandard2.0'">
<PackageReference Include="System.Buffers" Version="4.5.1" />
<PackageReference Include="System.Memory" Version="4.5.5" />
</ItemGroup>
<ItemGroup>
<None Include="..\CHANGELOG.md">
<Link>CHANGELOG.md</Link>
</None>
</ItemGroup>
</Project>

64
binding/csharp/IP2Region.Net/Internal/Abstractions/AbstractCacheStrategy.cs
View File

@ -1,64 +0,0 @@
// Copyright 2023 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Alan <lzh.shap@gmail.com>
// @Date 2023/07/25
using System.Buffers;
namespace IP2Region.Net.Internal.Abstractions;
internal abstract class AbstractCacheStrategy
{
protected const int HeaderInfoLength = 256;
protected const int VectorIndexRows = 256;
protected const int VectorIndexCols = 256;
protected const int VectorIndexSize = 8;
protected readonly FileStream XdbFileStream;
private const int BufferSize = 4096;
internal int IoCount { get; private set; }
protected AbstractCacheStrategy(string xdbPath)
{
XdbFileStream = new FileStream(xdbPath, FileMode.Open, FileAccess.Read, FileShare.Read, BufferSize,
useAsync: true);
}
protected int GetVectorIndexStartPos(uint ip)
{
var il0 = ip >> 24 & 0xFF;
var il1 = ip >> 16 & 0xFF;
var idx = il0 * VectorIndexCols * VectorIndexSize + il1 * VectorIndexSize;
return (int)idx;
}
internal abstract ReadOnlyMemory<byte> GetVectorIndex(uint ip);
internal virtual ReadOnlyMemory<byte> GetData(int offset, int length)
{
byte[] buffer = ArrayPool<byte>.Shared.Rent(length);
int totalBytesRead = 0;
try
{
XdbFileStream.Seek(offset, SeekOrigin.Begin);
int bytesRead;
do
{
int bytesToRead = Math.Min(BufferSize, length - totalBytesRead);
bytesRead = XdbFileStream.Read(buffer, totalBytesRead, bytesToRead);
totalBytesRead += bytesRead;
IoCount++;
} while (bytesRead > 0 && totalBytesRead < length);
}
finally
{
ArrayPool<byte>.Shared.Return(buffer);
}
return new ReadOnlyMemory<byte>(buffer, 0, totalBytesRead);
}
}

31
binding/csharp/IP2Region.Net/Internal/CacheStrategyFactory.cs
View File

@ -1,31 +0,0 @@
// Copyright 2023 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Alan <lzh.shap@gmail.com>
// @Date 2023/07/25
using IP2Region.Net.Internal.Abstractions;
using IP2Region.Net.XDB;
namespace IP2Region.Net.Internal;
internal class CacheStrategyFactory
{
private readonly string _xdbPath;
public CacheStrategyFactory(string xdbPath)
{
_xdbPath = xdbPath;
}
public AbstractCacheStrategy CreateCacheStrategy(CachePolicy cachePolicy)
{
return cachePolicy switch
{
CachePolicy.Content => new ContentCacheStrategy(_xdbPath),
CachePolicy.VectorIndex => new VectorIndexCacheStrategy(_xdbPath),
CachePolicy.File => new FileCacheStrategy(_xdbPath),
_ => throw new ArgumentException(nameof(cachePolicy))
};
}
}

32
binding/csharp/IP2Region.Net/Internal/ContentCacheStrategy.cs
View File

@ -1,32 +0,0 @@
// Copyright 2023 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Alan <lzh.shap@gmail.com>
// @Date 2023/07/25
using IP2Region.Net.Internal.Abstractions;
namespace IP2Region.Net.Internal;
internal class ContentCacheStrategy : AbstractCacheStrategy
{
private readonly ReadOnlyMemory<byte> _cacheData;
public ContentCacheStrategy(string xdbPath) : base(xdbPath)
{
_cacheData = base.GetData(0, (int)XdbFileStream.Length);
XdbFileStream.Close();
XdbFileStream.Dispose();
}
internal override ReadOnlyMemory<byte> GetVectorIndex(uint ip)
{
int idx = GetVectorIndexStartPos(ip);
return _cacheData.Slice(HeaderInfoLength + idx, VectorIndexSize);
}
internal override ReadOnlyMemory<byte> GetData(int offset, int length)
{
return _cacheData.Slice(offset, length);
}
}

22
binding/csharp/IP2Region.Net/Internal/FileCacheStrategy.cs
View File

@ -1,22 +0,0 @@
// Copyright 2023 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Alan <lzh.shap@gmail.com>
// @Date 2023/07/25
using IP2Region.Net.Internal.Abstractions;
namespace IP2Region.Net.Internal;
internal class FileCacheStrategy : AbstractCacheStrategy
{
public FileCacheStrategy(string xdbPath) : base(xdbPath)
{
}
internal override ReadOnlyMemory<byte> GetVectorIndex(uint ip)
{
var idx = GetVectorIndexStartPos(ip);
return GetData(HeaderInfoLength + idx, VectorIndexSize);
}
}

26
binding/csharp/IP2Region.Net/Internal/VectorIndexCacheStrategy.cs
View File

@ -1,26 +0,0 @@
// Copyright 2023 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Alan <lzh.shap@gmail.com>
// @Date 2023/07/25
using IP2Region.Net.Internal.Abstractions;
namespace IP2Region.Net.Internal;
internal class VectorIndexCacheStrategy : AbstractCacheStrategy
{
private readonly ReadOnlyMemory<byte> _vectorIndex;
public VectorIndexCacheStrategy(string xdbPath) : base(xdbPath)
{
var vectorLength = VectorIndexRows * VectorIndexCols * VectorIndexSize;
_vectorIndex = base.GetData(HeaderInfoLength, vectorLength);
}
internal override ReadOnlyMemory<byte> GetVectorIndex(uint ip)
{
var idx = GetVectorIndexStartPos(ip);
return _vectorIndex.Slice(idx, VectorIndexSize);
}
}

17
binding/csharp/IP2Region.Net/XDB/CachePolicy.cs
View File

@ -1,17 +0,0 @@
namespace IP2Region.Net.XDB;
public enum CachePolicy
{
/// <summary>
/// no cache
/// </summary>
File,
/// <summary>
/// cache vector index , reduce the number of IO operations
/// </summary>
VectorIndex,
/// <summary>
/// default cache policy , cache whole xdb file
/// </summary>
Content
}

85
binding/csharp/IP2Region.Net/XDB/Searcher.cs
View File

@ -1,85 +0,0 @@
// Copyright 2023 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Alan Lee <lzh.shap@gmail.com>
// @Date 2023/07/23
using System.Net;
using System.Runtime.InteropServices;
using System.Text;
using IP2Region.Net.Abstractions;
using IP2Region.Net.Internal;
using IP2Region.Net.Internal.Abstractions;
namespace IP2Region.Net.XDB;
public class Searcher : ISearcher
{
const int SegmentIndexSize = 14;
private readonly AbstractCacheStrategy _cacheStrategy;
public int IoCount => _cacheStrategy.IoCount;
public Searcher(CachePolicy cachePolicy, string dbPath)
{
var factory = new CacheStrategyFactory(dbPath);
_cacheStrategy = factory.CreateCacheStrategy(cachePolicy);
}
public string? Search(string ipStr)
{
var ip = Util.IpAddressToUInt32(ipStr);
return Search(ip);
}
public string? Search(IPAddress ipAddress)
{
var ip = Util.IpAddressToUInt32(ipAddress);
return Search(ip);
}
public string? Search(uint ip)
{
var index = _cacheStrategy.GetVectorIndex(ip);
uint sPtr = MemoryMarshal.Read<uint>(index.Span);
uint ePtr = MemoryMarshal.Read<uint>(index.Span.Slice(4));
var dataLen = 0;
uint dataPtr = 0;
uint l = 0;
uint h = (ePtr -sPtr) / SegmentIndexSize;
while (l <= h)
{
var mid = Util.GetMidIp(l, h);
var pos = sPtr + mid * SegmentIndexSize;
var buffer = _cacheStrategy.GetData((int)pos, SegmentIndexSize);
uint sip = MemoryMarshal.Read<uint>(buffer.Span);
uint eip = MemoryMarshal.Read<uint>(buffer.Span.Slice(4));
if (ip < sip)
{
h = mid - 1;
}
else if (ip > eip)
{
l = mid + 1;
}
else
{
dataLen = MemoryMarshal.Read<ushort>(buffer.Span.Slice(8));
dataPtr = MemoryMarshal.Read<uint>(buffer.Span.Slice(10));
break;
}
}
if (dataLen == 0)
{
return default;
}
var regionBuff = _cacheStrategy.GetData((int)dataPtr,dataLen);
return Encoding.UTF8.GetString(regionBuff.Span.ToArray());
}
}

23
binding/csharp/IP2Region.Net/XDB/Util.cs
View File

@ -1,23 +0,0 @@
using System.Net;
using System.Runtime.InteropServices;
namespace IP2Region.Net.XDB;
public static class Util
{
public static uint IpAddressToUInt32(string ipAddress)
{
var address = IPAddress.Parse(ipAddress);
return IpAddressToUInt32(address);
}
public static uint IpAddressToUInt32(IPAddress ipAddress)
{
byte[] bytes = ipAddress.GetAddressBytes();
Array.Reverse(bytes);
return MemoryMarshal.Read<uint>(bytes);
}
public static uint GetMidIp(uint x, uint y)
=> (x & y) + ((x ^ y) >> 1);
}

62
binding/csharp/README.md
View File

@ -1,62 +0,0 @@
# IP2Region.Net
.NET client library for IP2Region
## Installation
Install the package with [NuGet](https://www.nuget.org/packages/IP2Region.Net)
```bash
Install-Package IP2Region.Net
```
## Usage
```csharp
using IP2Region.Net.Abstractions;
using IP2Region.Net.XDB;
ISearcher searcher = new Searcher(CachePolicy , "your xdb file path");
```
### Cache Policy Description
| Cache Policy | Description | Thread Safe |
|-------------------------|------------------------------------------------------------------------------------------------------------|-------------|
| CachePolicy.Content | Cache the entire `xdb` data. | Yes |
| CachePolicy.VectorIndex | Cache `vecotorIndex` to speed up queries and reduce system io pressure by reducing one fixed IO operation. | Yes |
| CachePolicy.File | Completely file-based queries | Yes |
### XDB File Description
Generate using [maker](https://github.com/lionsoul2014/ip2region/tree/master/maker/csharp), or [download](https://github.com/lionsoul2014/ip2region/blob/master/data/ip2region.xdb) pre-generated xdb files
## ASP.NET Core Usage
```csharp
services.AddSingleton<ISearcher>(new Searcher(CachePolicy , "your xdb file path"));
```
## Performance
``` ini
BenchmarkDotNet=v0.13.2, OS=macOS 13.4.1 (c) (22F770820d) [Darwin 22.5.0]
Apple M1, 1 CPU, 8 logical and 8 physical cores
.NET SDK=7.0.306
[Host] : .NET 6.0.20 (6.0.2023.32017), Arm64 RyuJIT AdvSIMD
DefaultJob : .NET 6.0.20 (6.0.2023.32017), Arm64 RyuJIT AdvSIMD
```
| Method | Mean | Error | StdDev |
|-------------------------|-----------:|---------:|---------:|
| CachePolicy_Content | 155.7 ns | 0.46 ns | 0.39 ns |
| CachePolicy_File | 2,186.8 ns | 34.27 ns | 32.06 ns |
| CachePolicy_VectorIndex | 1,570.3 ns | 27.53 ns | 22.99 ns |
## Contributing
Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.
Please make sure to update tests as appropriate.
## License
[Apache License 2.0](https://github.com/lionsoul2014/ip2region/blob/master/LICENSE.md)

129
binding/erlang/README.md
View File

@ -1,129 +0,0 @@
# ip2region xdb erlang 查询客户端
### 简介
该bingding以erlang语言实现xdb查询客户端，基于Erlang OTP Application，查询逻辑由ip2region_worker工作进程实现，支持配多个工作进程来进行负载均衡。
### 应用配置
该应用可配置的参数在ip2region.app.src中,如下：
``` erlang
{env,[
{poolargs, [
{size, 1}, %% 工作进程默认数量
{max_overflow, 5} %% 工作进程最大数量
]}
]}
```
### 编译
```
$ rebar3 compile
```
### 运行
将xdb文件放到priv目录下，然后启动erlang节点：
```
$ rebar3 shell
```
在erlang shell中调用xdb:search/1接口查询Ip地址信息, 该接口支持以list格式字符串、binary格式字符串、tuple和整数表示的IP地址，如下：
```
1> xdb:search("1.0.8.0").
[20013,22269,124,48,124,24191,19996,30465,124,24191,24030,
24066,124,30005,20449]
2>
3> io:format("~ts~n", [xdb:search("1.0.8.0")]).
中国|0|广东省|广州市|电信
io:format("~ts~n", [xdb:search(<<"1.0.8.0">>)]).
中国|0|广东省|广州市|电信
4> io:format("~ts~n", [xdb:search({1,0,8,0})]).
中国|0|广东省|广州市|电信
6> io:format("~ts~n", [xdb:search(16779264)]).
中国|0|广东省|广州市|电信
```
### 使用方法
* 在rebar.config中引入依赖
```
{deps, [
ip2region
]}.
```
* 启动ip2region Application
```
......
application:ensure_started(ip2region),
......
```
* 调用xdb:search/1接口查询IP信息
```
......
ip2region:search("1.0.8.0"),
......
```
### 单元测试
```
$ rebar3 eunit
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling ip2region
===> Performing EUnit tests...
=INFO REPORT==== 17-Jan-2023::11:52:59.920155 ===
XdbFile:/home/admin/erl-workspace/ip2region/binding/erlang/_build/test/lib/ip2region/priv/ip2region.xdb
....
Finished in 0.074 seconds
4 tests, 0 failures
```
### 基准测试
```
$ cd benchmarks/
$ sh xdb-benchmark.sh
===> Verifying dependencies...
===> Analyzing applications...
===> Compiling ip2region
Erlang/OTP 24 [erts-12.3.2.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit]
Eshell V12.3.2.2 (abort with ^G)
1> =INFO REPORT==== 17-Jan-2023::11:37:35.631095 ===
XdbFile:/home/admin/erl-workspace/ip2region/binding/erlang/_build/default/lib/ip2region/priv/ip2region.xdb
===> Booted ip2region
===> Evaluating: "xdb_benchmark:main(\"../../data/ip.merge.txt\"), init:stop()."
CPU info:
model name : AMD EPYC 7K62 48-Core Processor
cache size : 512 KB
cpu MHz : 2595.124
bogomips : 5190.24
cores/threads : 2
Erlang info:
system_version:Erlang/OTP 24 [erts-12.3.2.2] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [jit]
load test data use 4.835593s
start run benchmark tests
search from file:
ip count:683844,
total time: 28.201699s,
search 24248.326315375536 times per second,
use 41.23995969841075 micro second per search
search from cache:
ip count:683844,
total time: 0.671801s,
search 1017926.4395259906 times per second,
use 0.9823892583688677 micro second per search
benchmark test finish
```

5
binding/erlang/benchmarks/xdb-benchmark.sh
View File

@ -1,5 +0,0 @@
#!/bin/bash
cd ..
rebar3 shell --eval="xdb_benchmark:main(\"../../data/ip.merge.txt\"), init:stop()."

26
binding/erlang/include/ip2region.hrl
View File

@ -1,26 +0,0 @@
-ifndef(IP2REGION_HRL).
-define(IP2REGION_HRL, true).
-define(NONE, none).
-define(APP_NAME, ip2region).
-define(XDB_VECTOR_INDEX, ets_xdb_vector_index).
-define(XDB_SEGMENT_INDEX, ets_xdb_segement_index).
-define(IP2REGION_CACHE, ets_ip2region_cache).
-define(XDB_HEADER_SIZE, 256).
-define(XDB_VECTOR_COLS, 256).
-define(XDB_VECTOR_INDEX_SIZE, 8).
-define(XDB_VECTOR_INDEX_COUNT, (16#10000)). %% 256*256
-define(XDB_SEGMENT_INDEX_SIZE, 14).
-define(IP2REGION_POOL, ip2region_pool).
-ifndef(IF).
-define(IF(C, T, F), case (C) of true -> (T); false -> (F) end).
-define(IF(C, T), ?IF(C, T, skip)).
-endif.
-endif.

0
binding/erlang/priv/dummy
View File

24
binding/erlang/rebar.config
View File

@ -1,24 +0,0 @@
{erl_opts, [
debug_info,
export_all,
nowarn_export_all
]}.
{plugins, [rebar3_hex, rebar3_ex_doc]}.
{deps, [
poolboy
]}.
{shell, [
% {config, "config/sys.config"},
{apps, [ip2region]}
]}.
{ex_doc, [
{extras, ["README.md"]},
{main, "README.md"},
{source_url, "https://github.com/leihua996/ip2region/tree/master/binding/erlang"}
]}.
{hex, [{doc, ex_doc}]}.

8
binding/erlang/rebar.lock
View File

@ -1,8 +0,0 @@
{"1.2.0",
[{<<"poolboy">>,{pkg,<<"poolboy">>,<<"1.5.2">>},0}]}.
[
{pkg_hash,[
{<<"poolboy">>, <<"392B007A1693A64540CEAD79830443ABF5762F5D30CF50BC95CB2C1AAAFA006B">>}]},
{pkg_hash_ext,[
{<<"poolboy">>, <<"DAD79704CE5440F3D5A3681C8590B9DC25D1A561E8F5A9C995281012860901E3">>}]}
].

20
binding/erlang/src/ip2region.app.src
View File

@ -1,20 +0,0 @@
{application, ip2region,
[{description, "ip2region xdb client application"},
{vsn, "0.1.0"},
{registered, []},
{mod, {ip2region_app, []}},
{applications,
[kernel,
stdlib
]},
{env,[
{poolargs, [
{size, 1},
{max_overflow, 5}
]}
]},
{modules, []},
{licenses, ["Apache-2.0"]},
{links, [{"Github", "https://github.com/leihua996/ip2region/tree/master/binding/erlang"}]}
]}.

22
binding/erlang/src/ip2region_app.erl
View File

@ -1,22 +0,0 @@
%%%-------------------------------------------------------------------
%% Copyright 2022 The Ip2Region Authors. All rights reserved.
%% Use of this source code is governed by a Apache2.0-style
%% license that can be found in the LICENSE file.
%%
%% @doc
%% @end
%%%-------------------------------------------------------------------
-module(ip2region_app).
-behaviour(application).
-export([start/2, stop/1]).
start(_StartType, _StartArgs) ->
ip2region_sup:start_link().
stop(_State) ->
ok.
%% internal functions

55
binding/erlang/src/ip2region_sup.erl
View File

@ -1,55 +0,0 @@
%%%-------------------------------------------------------------------
%% Copyright 2022 The Ip2Region Authors. All rights reserved.
%% Use of this source code is governed by a Apache2.0-style
%% license that can be found in the LICENSE file.
%%
%% @doc ip2region top level supervisor.
%% @end
%%%-------------------------------------------------------------------
-module(ip2region_sup).
-behaviour(supervisor).
-include("ip2region.hrl").
-export([start_link/0]).
-export([init/1, create_table/0]).
-define(SERVER, ?MODULE).
start_link() ->
{ok, SupPid} = supervisor:start_link({local, ?SERVER}, ?MODULE, []),
{ok, _PoolPid} = start_ip2region_pool(SupPid),
{ok, SupPid}.
%% sup_flags() = #{strategy => strategy(), % optional
%% intensity => non_neg_integer(), % optional
%% period => pos_integer()} % optional
%% child_spec() = #{id => child_id(), % mandatory
%% start => mfargs(), % mandatory
%% restart => restart(), % optional
%% shutdown => shutdown(), % optional
%% type => worker(), % optional
%% modules => modules()} % optional
init([]) ->
create_table(),
SupFlags = #{strategy => one_for_one,
intensity => 10,
period => 5},
ChildSpecs = [],
{ok, {SupFlags, ChildSpecs}}.
%% internal functions
%%
create_table() ->
Opts = [named_table, set, public, {read_concurrency, true}, {keypos, 1}],
ets:new(?XDB_VECTOR_INDEX, Opts),
ets:new(?XDB_SEGMENT_INDEX, Opts),
ets:new(?IP2REGION_CACHE, Opts).
start_ip2region_pool(Sup) ->
{ok, PoolArgsCfg} = application:get_env(poolargs),
PoolName = ?IP2REGION_POOL,
PoolArgs = [{strategy, fifo}, {name, {local, PoolName}}, {worker_module, ip2region_worker} | PoolArgsCfg],
WorkerArgs = [],
ChildSpecs = poolboy:child_spec(PoolName, PoolArgs, WorkerArgs),
supervisor:start_child(Sup, ChildSpecs).

26
binding/erlang/src/ip2region_util.erl
View File

@ -1,26 +0,0 @@
%%%-------------------------------------------------------------------
%% Copyright 2022 The Ip2Region Authors. All rights reserved.
%% Use of this source code is governed by a Apache2.0-style
%% license that can be found in the LICENSE file.
%%
%% @doc
%% ip2region utils
%% @end
%%%-------------------------------------------------------------------
-module(ip2region_util).
-export([ipv4_to_n/1]).
ipv4_to_n(IntIp) when is_integer(IntIp) -> IntIp;
ipv4_to_n({A, B, C, D}) ->
<<N:32>> = <<A, B, C, D>>,
N;
ipv4_to_n(Ip) when is_binary(Ip) ->
ipv4_to_n(binary_to_list(Ip));
ipv4_to_n(Ip) when is_list(Ip) ->
case inet_parse:address(Ip) of
{ok, Addr} ->
ipv4_to_n(Addr);
_ ->
{error, bad_ip_format}
end.

183
binding/erlang/src/ip2region_worker.erl
View File

@ -1,183 +0,0 @@
%%%-------------------------------------------------------------------
%% Copyright 2022 The Ip2Region Authors. All rights reserved.
%% Use of this source code is governed by a Apache2.0-style
%% license that can be found in the LICENSE file.
%%
%% @doc
%% ip2region xdb client worker
%% @end
%%%-------------------------------------------------------------------
-module(ip2region_worker).
-behaviour(gen_server).
-include("ip2region.hrl").
%% API
-export([start/1, stop/1, start_link/1]).
-export([search/2]).
-export([init/1, handle_call/3, handle_cast/2, handle_info/2, terminate/2, code_change/3]).
-record(state, {xdb_fd}).
%%==========================================
%% API
%% =========================================
start(Args) ->
Opts = [{spawn_opt, [{min_heap_size, 6000}]}],
gen_server:start(?MODULE, Args, Opts).
start_link(Args) ->
Opts = [{spawn_opt, [{min_heap_size, 6000}]}],
gen_server:start_link(?MODULE, Args, Opts).
stop(Pid) ->
gen_server:call(Pid, stop).
search(Pid, Ip) ->
gen_server:call(Pid, {search, Ip}).
%%==========================================
%% gen_server callbacks
%% =========================================
init(_Args) ->
process_flag(trap_exit, true),
AppName =
case application:get_application() of
{ok, AName} -> AName;
_ -> ?APP_NAME
end,
PrivDir = code:priv_dir(AppName),
XdbFileName = filename:join([PrivDir, "ip2region.xdb"]),
error_logger:info_report(io_lib:format("XdbFile:~s~n", [XdbFileName])),
{ok, IoDevice} = file:open(XdbFileName, [read, binary]),
load_vector_index(IoDevice),
{ok, #state{xdb_fd = IoDevice}}.
handle_call(Request, From, State) ->
try
do_call(Request, From, State)
catch
Class:Error:Stacktrace ->
error_logger:error_report(io_lib:format("~p handle call error, Req:~p ~p, stacktrace:~p~n",
[?MODULE, Request, {Class, Error}, Stacktrace])),
{reply, {error, {Class, Error}}, State}
end.
handle_cast(Msg, State) ->
try
do_cast(Msg, State)
catch
Class:Error:Stacktrace ->
error_logger:error_report(io_lib:format("~p handle cast error, Msg:~p, ~p, stacktrace:~w~n",
[?MODULE, Msg, {Class, Error}, Stacktrace])),
{noreply, State}
end.
handle_info(Info, State) ->
try
do_info(Info, State)
catch
Class:Error:Stacktrace ->
error_logger:error_report(io_lib:format("~p handle info error, Info:~p, ~p, stacktrace:~p~n",
[?MODULE, Info, {Class, Error}, Stacktrace])),
{noreply, State}
end.
terminate(_Reason, State) ->
#state{xdb_fd = XdbFd} = State,
case is_pid(XdbFd) of
true ->
file:close(XdbFd);
_ ->
skip
end,
ok.
code_change(_OldVsn, State, _Extra) ->
{ok, State}.
%%==========================================
%% Internal function
%% =========================================
do_call({search, Ip}, _From, #state{xdb_fd = IoDevice} = State) ->
Reply = search_ip(IoDevice, Ip),
{reply, Reply, State};
do_call(stop, _From, State) ->
{stop, normal, stopped, State};
do_call(Request, From, State) ->
error_logger:error_report(io_lib:format("unknown request: ~p, from:~p", [Request, From])),
{noreply, State}.
do_cast(Msg, State) ->
error_logger:error_report(io_lib:format("unknown msg: ~p", [Msg])),
{noreply, State}.
do_info(Info, State) ->
error_logger:error_report(io:format("unknown info: ~p", [Info])),
{noreply, State}.
load_vector_index(IoDevice) ->
Key = ip2region_header_loaded,
case persistent_term:get(Key, false) of
true -> ok;
_ ->
{ok, <<_Header:?XDB_HEADER_SIZE/binary, VectorIndexBin/binary>> } =
file:read(IoDevice, ?XDB_HEADER_SIZE + ?XDB_VECTOR_INDEX_COUNT*8),
load_vector_index_aux(VectorIndexBin, 0),
persistent_term:put(Key, true)
end.
load_vector_index_aux(<<>>, _Index) -> ok;
load_vector_index_aux(<<SPtr:32/little, EPtr:32/little, VectorIndexBin/binary>>, Index) ->
Term = {Index, SPtr, EPtr},
ets:insert(?XDB_VECTOR_INDEX, Term),
load_vector_index_aux(VectorIndexBin, Index + 1).
search_ip(IoDevice, Ip) ->
IntIp = ip2region_util:ipv4_to_n(Ip),
case ets:lookup(?IP2REGION_CACHE, IntIp) of
[{_IntIp, RegionInfo}] ->
RegionInfo;
_ ->
<<A:8, B:8, _Rest/binary>> = <<IntIp:32>>,
VectorIdx = A * ?XDB_VECTOR_COLS + B,
[{_, SPtr, EPtr}] = ets:lookup(?XDB_VECTOR_INDEX, VectorIdx),
RegionInfo = search_ip(IoDevice, IntIp, SPtr, EPtr, 0, (EPtr - SPtr) div ?XDB_SEGMENT_INDEX_SIZE),
ets:insert_new(?IP2REGION_CACHE, {IntIp, RegionInfo}),
RegionInfo
end.
search_ip(IoDevice, IntIp, SPtr, EPtr, Low, High) when Low =< High ->
Middle = (Low + High) bsr 1,
SPtr2 = SPtr + Middle * ?XDB_SEGMENT_INDEX_SIZE,
{SIp, EIp, DataLen, DataPtr} = read_segement_index(IoDevice, SPtr2),
if
IntIp < SIp ->
search_ip(IoDevice, IntIp, SPtr, EPtr, Low, Middle - 1);
IntIp > EIp ->
search_ip(IoDevice, IntIp, SPtr, EPtr, Middle + 1, High);
true ->
{ok, DataBin} = read_file(IoDevice, DataPtr, DataLen),
unicode:characters_to_nfc_list(DataBin)
end;
search_ip(_IoDevice, _IntIp, _SPtr, _EPtr, _Low, _High) ->
{error, unknown}.
read_file(IoDevice, Position, DataLength) ->
file:position(IoDevice, {bof, Position}),
file:read(IoDevice, DataLength).
read_segement_index(IoDevice, SPtr) ->
case ets:lookup(?XDB_SEGMENT_INDEX, SPtr) of
[{_SPtr, SIp, EIp, DataLen, DataPtr}] ->
{SIp, EIp, DataLen, DataPtr};
_ ->
{ok, <<SIp:32/little, EIp:32/little, DataLen:16/little, DataPtr:32/little>>} =
read_file(IoDevice, SPtr, ?XDB_SEGMENT_INDEX_SIZE),
ets:insert_new(?XDB_SEGMENT_INDEX, {SPtr, SIp, EIp, DataLen, DataPtr}),
{SIp, EIp, DataLen, DataPtr}
end.

32
binding/erlang/src/xdb.erl
View File

@ -1,32 +0,0 @@
%%%-------------------------------------------------------------------
%% Copyright 2022 The Ip2Region Authors. All rights reserved.
%% Use of this source code is governed by a Apache2.0-style
%% license that can be found in the LICENSE file.
%%
%% @doc
%% ip2region xdb client search api
%% @end
%%%-------------------------------------------------------------------
-module(xdb).
-include("ip2region.hrl").
-export([search/1]).
-spec search(Ip :: tuple() | list() | binary()) -> Result :: binary | {error, Reason::atom()}.
search(Ip) when is_integer(Ip); is_list(Ip); is_tuple(Ip); is_binary(Ip) ->
case ip2region_util:ipv4_to_n(Ip) of
IntIp when is_integer(IntIp) ->
case ets:lookup(?IP2REGION_CACHE, IntIp) of
[{_IntIp, Region}] -> Region;
_ ->
Worker = poolboy:checkout(?IP2REGION_POOL, true, infinity),
try
ip2region_worker:search(Worker, IntIp)
after
poolboy:checkin(?IP2REGION_POOL, Worker)
end
end;
Ret ->
Ret
end.

75
binding/erlang/src/xdb_benchmark.erl
View File

@ -1,75 +0,0 @@
%%%-------------------------------------------------------------------
%% Copyright 2022 The Ip2Region Authors. All rights reserved.
%% Use of this source code is governed by a Apache2.0-style
%% license that can be found in the LICENSE file.
%%
%% @doc
%% ip2region xdb client benchmark test
%% @end
%%%-------------------------------------------------------------------
-module(xdb_benchmark).
-export([main/1]).
main(DataFile) ->
application:ensure_started(ip2region),
show_hw_sw_info(),
IpList = load_test_data(DataFile),
run(IpList).
show_hw_sw_info() ->
io:format("CPU info:~n", []),
io:format("~s", [os:cmd("egrep '^model name' /proc/cpuinfo | head -1")]),
io:format("~s", [os:cmd("egrep '^cache' /proc/cpuinfo | head -1")]),
io:format("~s", [os:cmd("egrep '^cpu MHz' /proc/cpuinfo | head -1")]),
io:format("~s", [os:cmd("egrep '^bogomips' /proc/cpuinfo | head -1")]),
io:format("cores/threads : ~s~n", [os:cmd("egrep -c '^processor' /proc/cpuinfo")]),
io:format("Erlang info:~n", []),
io:format("system_version:~s", [erlang:system_info(system_version)]),
ok.
load_test_data(DataFile) ->
{ok, Fd} = file:open(DataFile, [read]),
T0 = os:timestamp(),
IpList = load_test_data(Fd, []),
T1 = os:timestamp(),
Sec = timer:now_diff(T1, T0) / 1000000,
io:format("load test data use ~ps~n", [Sec]),
IpList.
load_test_data(Fd, IpList) ->
case file:read_line(Fd) of
{ok, Ip} ->
case string:tokens(unicode:characters_to_list(Ip), "|") of
[SIp | _Tail] ->
load_test_data(Fd, [string:trim(SIp)| IpList]);
_ ->
load_test_data(Fd, IpList)
end;
_ ->
file:close(Fd),
IpList
end.
run(IpList) ->
garbage_collect(),
io:format("~nstart run benchmark tests~n", []),
io:format("~nsearch from file:~n", []),
run_test(IpList),
io:format("~nsearch from cache:~n", []),
run_test(IpList),
io:format("~nbenchmark test finish~n", []).
run_test(IpList) ->
T0 = os:timestamp(),
run_test_aux(IpList),
T1 = os:timestamp(),
Sec = timer:now_diff(T1, T0) / 1000000,
IpCount = length(IpList),
io:format("ip count:~p,~ntotal time: ~ps,~nsearch ~p times per second,~nuse ~p micro second per search~n",
[IpCount, Sec, IpCount / Sec, Sec * 1000000/IpCount]).
run_test_aux([]) -> ok;
run_test_aux([Ip | Tail]) ->
xdb:search(Ip),
run_test_aux(Tail).

18
binding/erlang/test/xdb_test.erl
View File

@ -1,18 +0,0 @@
-module(xdb_test).
-include_lib("eunit/include/eunit.hrl").
search_test_() ->
application:ensure_started(ip2region),
A = "中国|0|广东省|广州市|电信",
Region0 = xdb:search("1.0.8.0"),
Region1 = xdb:search(<<"1.0.8.0">>),
Region2 = xdb:search({1,0,8,0}),
Region3 = xdb:search("xxx.0.8.0"),
[
?_assert(A =:= Region0),
?_assert(A =:= Region1),
?_assert(A =:= Region2),
?_assert({error, bad_ip_format} =:= Region3)
].

10
binding/golang/Makefile
View File

@ -1,10 +0,0 @@
# ip2region golang binding makefile
all: build
.PHONY: all
build:
go build -o xdb_searcher
test:
go test -v ./...
clean:
find ./ -name xdb_searcher | xargs rm -f

190
binding/golang/ReadMe.md
View File

@ -1,190 +0,0 @@
# ip2region xdb golang 查询客户端实现
# 使用方式
### package 获取
```bash
go get github.com/lionsoul2014/ip2region/binding/golang
```
### 关于 IPv4 / IPv6
该 xdb 查询客户端实现同时支持对 IPv4 和 IPv6 的查询，使用方式如下：
```golang
// 如果是 IPv4: 设置 xdb 路径为 v4 的 xdb 文件，IP版本指定为 xdb.IPv4
dbPath := "../../data/ip2region_v4.xdb" // 或者你的 ipv4 xdb 的路径
version := xdb.IPv4
// 如果是 IPv6: 设置 xdb 路径为 v6 的 xdb 文件，IP版本指定为 xdb.IPv6
dbPath = "../../data/ip2region_v6.xdb" // 或者你的 ipv6 xdb 路径
version = xdb.IPv6
// dbPath 指定的 xdb 的 IP 版本必须和 version 指定的一致，不然查询执行的时候会报错
// 备注：以下演示直接使用 dbPath 和 version 变量
```
### 文件验证
建议您主动去验证 xdb 文件的适用性，因为后期的一些新功能可能会导致目前的 Searcher 版本无法适用你使用的 xdb 文件，验证可以避免运行过程中的一些不可预测的错误。
你不需要每次都去验证，例如在服务启动的时候，或者手动调用命令验证确认版本匹配即可，不要在每次创建的 Searcher 的时候运行验证，这样会影响查询的响应速度，尤其是高并发的使用场景。
```golang
err := xdb.VerifyFromFile(dbPath)
if err != nil {
// err 包含的验证的错误
return fmt.Errorf("xdb file verify: %w", err)
}
// 当前使用的 Searcher 可以安全的用于对 dbPath 指向的 xdb 的查询操作
```
### 完全基于文件的查询
```golang
import (
"fmt"
"github.com/lionsoul2014/ip2region/binding/golang/xdb"
"time"
)
func main() {
// 通过 version 和 dbPath 创建完全基于文件的查询对象
searcher, err := xdb.NewWithFileOnly(version, dbPath)
if err != nil {
fmt.Printf("failed to create searcher: %s\n", err.Error())
return
}
defer searcher.Close()
// 定位信息查询：IPv4 或者 IPv6 的地址都支持
var ip = "1.2.3.4" // IPv4
// ip = "2001:4:112:ffff:ffff:ffff:ffff:ffff" // IPv6
var tStart = time.Now()
region, err := searcher.SearchByStr(ip)
if err != nil {
fmt.Printf("failed to SearchIP(%s): %s\n", ip, err)
return
}
// IPv4 或者 IPv6 的定位信息
fmt.Printf("{region: %s, took: %s}\n", region, time.Since(tStart))
// 备注：并发使用，每个 goroutine 需要创建一个独立的 searcher 对象。
}
```
### 缓存 `VectorIndex` 索引
可以预先加载 `vectorIndex` 缓存，然后做成全局变量，每次创建 searcher 的时候使用全局的 `vectorIndex`，可以减少一次固定的 IO 操作从而加速查询，减少系统 io 压力。
```golang
// 1、从 dbPath 加载 VectorIndex 缓存，把下述 vIndex 变量全局到内存里面。
vIndex, err := xdb.LoadVectorIndexFromFile(dbPath)
if err != nil {
fmt.Printf("failed to load vector index from `%s`: %s\n", dbPath, err)
return
}
// 2、用全局的 vIndex 创建带 VectorIndex 缓存的查询对象。
searcher, err := xdb.NewWithVectorIndex(version, dbPath, vIndex)
if err != nil {
fmt.Printf("failed to create searcher with vector index: %s\n", err)
return
}
// 备注：并发使用，全部 goroutine 共享全局的只读 vIndex 缓存，每个 goroutine 创建一个独立的 searcher 对象
```
### 缓存整个 `xdb` 数据
可以预先加载整个 ip2region.xdb 到内存，完全基于内存查询，类似于之前的 memory search 查询。
```golang
// 1、从 dbPath 加载整个 xdb 到内存
cBuff, err := xdb.LoadContentFromFile(dbPath)
if err != nil {
fmt.Printf("failed to load content from `%s`: %s\n", dbPath, err)
return
}
// 2、用全局的 cBuff 创建完全基于内存的查询对象。
searcher, err := xdb.NewWithBuffer(version, cBuff)
if err != nil {
fmt.Printf("failed to create searcher with content: %s\n", err)
return
}
// 备注：并发使用，用整个 xdb 缓存创建的 searcher 对象可以安全用于并发。
```
# 编译测试程序
通过如下方式编译得到 xdb_searcher 可执行程序：
```bash
# 切换到 golang binding 根目录
make
```
# 查询测试
通过 `xdb_searcher search` 命令来测试 xdb 的查询：
```bash
➜ golang git:(fr_xdb_ipv6) ✗ ./xdb_searcher search
./xdb_searcher search [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：使用默认的 data/ip2region_v4.xdb 进行 IPv4 的查询测试
```bash
➜ golang git:(master) ./xdb_searcher search --db=../../data/ip2region_v4.xdb
ip2region xdb searcher test program
source xdb: ../../data/ip2region_v4.xdb (IPv4, vectorIndex)
type 'quit' to exit
ip2region>> 219.133.111.87
{region: 中国|广东省|深圳市|电信, ioCount: 2, took: 19.005µs}
ip2region>>
```
例如：使用默认的 data/ip2region_v6.xdb 进行 IPv6 的查询：
```bash
➜ golang git:(master) ✗ ./xdb_searcher search --db=../../data/ip2region_v6.xdb
ip2region xdb searcher test program
source xdb: ../../data/ip2region_v6.xdb (IPv6, vectorIndex)
type 'quit' to exit
ip2region>> 240e:87c:892:ffff:ffff:ffff:ffff:ffff
{region: 中国|广东省|深圳市|专线用户, ioCount: 12, took: 130.294µs}
ip2region>>
```
输入对应版本的 ip 地址进行查询即可，输入 quit 退出测试程序。可以设置 `cache-policy` 为 file/vectorIndex/content 来测试不同的查询缓存机制。
# bench 测试
通过 `xdb_searcher bench` 命令来进行自动 bench 测试，一方面确保程序和 `xdb` 文件都没有错误，另一方面通过大量的查询得到平均查询性能：
```bash
➜ golang git:(fr_xdb_ipv6) ./xdb_searcher bench
./xdb_searcher bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：通过 data/ip2region_v4.xdb 和 data/ipv4_source.txt 进行 ipv4 的 bench 测试：
```bash
./xdb_searcher bench --db=../../data/ip2region_v4.xdb --src=../../data/ipv4_source.txt
```
例如：通过 data/ip2region_v6.xdb 和 data/ipv6_source.txt 进行 ipv6 的 bench 测试：
```bash
./xdb_searcher bench --db=../../data/ip2region_v6.xdb --src=../../data/ipv6_source.txt
```
可以设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 不同缓存实现机制的效率。
*请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件的相同的源文件*。
bench 程序会逐行读取 `src` 指定的源IP文件，然后每个 IP 段选取 5 个固定位置的 IP 进行测试，以确保查询的 region 信息和原始的 region 信息是相同。测试途中没有调试信息的输出，有错误会打印错误信息并且终止运行，所以看到 `Bench finished` 就表示 bench 成功了，cost 是表示每次查询操作的平均时间(ns)。

5
binding/golang/go.mod
View File

@ -1,5 +0,0 @@
module github.com/lionsoul2014/ip2region/binding/golang
go 1.17
require github.com/mitchellh/go-homedir v1.1.0

2
binding/golang/go.sum
View File

@ -1,2 +0,0 @@
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=

298
binding/golang/main.go
View File

@ -1,298 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package main
import (
"bufio"
"fmt"
"log"
"os"
"strings"
"time"
"github.com/lionsoul2014/ip2region/binding/golang/xdb"
"github.com/mitchellh/go-homedir"
)
func printHelp() {
fmt.Printf("ip2region xdb searcher\n")
fmt.Printf("%s [command] [command options]\n", os.Args[0])
fmt.Printf("Command: \n")
fmt.Printf(" search search input test\n")
fmt.Printf(" bench search bench test\n")
}
func testSearch() {
var err error
var dbFile, cachePolicy = "", "vectorIndex"
for i := 2; i < len(os.Args); i++ {
r := os.Args[i]
if len(r) < 5 {
continue
}
if strings.Index(r, "--") != 0 {
continue
}
var sIdx = strings.Index(r, "=")
if sIdx < 0 {
fmt.Printf("missing = for args pair '%s'\n", r)
return
}
switch r[2:sIdx] {
case "db":
dbFile = r[sIdx+1:]
case "cache-policy":
cachePolicy = r[sIdx+1:]
default:
fmt.Printf("undefined option `%s`\n", r)
return
}
}
if dbFile == "" {
fmt.Printf("%s search [command options]\n", os.Args[0])
fmt.Printf("options:\n")
fmt.Printf(" --db string ip2region binary xdb file path\n")
fmt.Printf(" --cache-policy string cache policy: file/vectorIndex/content\n")
return
}
dbPath, err := homedir.Expand(dbFile)
if err != nil {
fmt.Printf("invalid xdb file path `%s`: %s", dbFile, err)
return
}
// create the searcher with the cache policy setting
searcher, err := createSearcher(dbPath, cachePolicy)
if err != nil {
fmt.Printf("failed to create searcher: %s\n", err.Error())
return
}
defer func() {
searcher.Close()
fmt.Printf("searcher test program exited, thanks for trying\n")
}()
fmt.Printf(`ip2region xdb searcher test program
source xdb: %s (%s, %s)
type 'quit' to exit
`, dbPath, searcher.IPVersion().Name, cachePolicy)
reader := bufio.NewReader(os.Stdin)
for {
fmt.Print("ip2region>> ")
str, err := reader.ReadString('\n')
if err != nil {
log.Fatalf("failed to read string: %s", err)
}
line := strings.TrimSpace(strings.TrimSuffix(str, "\n"))
if len(line) == 0 {
continue
}
if line == "quit" {
break
}
tStart := time.Now()
region, err := searcher.SearchByStr(line)
if err != nil {
fmt.Printf("\x1b[0;31m{err: %s, ioCount: %d}\x1b[0m\n", err.Error(), searcher.GetIOCount())
} else {
fmt.Printf("\x1b[0;32m{region: %s, ioCount: %d, took: %s}\x1b[0m\n", region, searcher.GetIOCount(), time.Since(tStart))
}
}
}
func testBench() {
var err error
var dbFile, srcFile, cachePolicy = "", "", "vectorIndex"
for i := 2; i < len(os.Args); i++ {
r := os.Args[i]
if len(r) < 5 {
continue
}
if strings.Index(r, "--") != 0 {
continue
}
var sIdx = strings.Index(r, "=")
if sIdx < 0 {
fmt.Printf("missing = for args pair '%s'\n", r)
return
}
switch r[2:sIdx] {
case "db":
dbFile = r[sIdx+1:]
case "src":
srcFile = r[sIdx+1:]
case "cache-policy":
cachePolicy = r[sIdx+1:]
default:
fmt.Printf("undefined option `%s`\n", r)
return
}
}
if dbFile == "" || srcFile == "" {
fmt.Printf("%s bench [command options]\n", os.Args[0])
fmt.Printf("options:\n")
fmt.Printf(" --db string ip2region binary xdb file path\n")
fmt.Printf(" --src string source ip text file path\n")
fmt.Printf(" --cache-policy string cache policy: file/vectorIndex/content\n")
return
}
dbPath, err := homedir.Expand(dbFile)
if err != nil {
fmt.Printf("invalid xdb file path `%s`: %s", dbFile, err)
return
}
searcher, err := createSearcher(dbPath, cachePolicy)
if err != nil {
fmt.Printf("failed to create searcher: %s\n", err.Error())
return
}
defer func() {
searcher.Close()
}()
handle, err := os.OpenFile(srcFile, os.O_RDONLY, 0600)
if err != nil {
fmt.Printf("failed to open source text file: %s\n", err)
return
}
defer handle.Close()
var count, tStart, costs = int64(0), time.Now(), int64(0)
var scanner = bufio.NewScanner(handle)
scanner.Split(bufio.ScanLines)
for scanner.Scan() {
var l = strings.TrimSpace(strings.TrimSuffix(scanner.Text(), "\n"))
var ps = strings.SplitN(l, "|", 3)
if len(ps) != 3 {
fmt.Printf("invalid ip segment line `%s`\n", l)
return
}
sip, err := xdb.ParseIP(ps[0])
if err != nil {
fmt.Printf("check start ip `%s`: %s\n", ps[0], err)
return
}
eip, err := xdb.ParseIP(ps[1])
if err != nil {
fmt.Printf("check end ip `%s`: %s\n", ps[1], err)
return
}
if xdb.IPCompare(sip, eip) > 0 {
fmt.Printf("start ip(%s) should not be greater than end ip(%s)\n", ps[0], ps[1])
return
}
for _, ip := range [][]byte{sip, eip} {
sTime := time.Now()
region, err := searcher.Search(ip)
if err != nil {
fmt.Printf("failed to search ip '%s': %s\n", xdb.IP2String(ip), err)
return
}
costs += time.Since(sTime).Nanoseconds()
// check the region info
if region != ps[2] {
fmt.Printf("failed Search(%s) with (%s != %s)\n", xdb.IP2String(ip), region, ps[2])
return
}
count++
}
}
cost := time.Since(tStart)
fmt.Printf("Bench finished, {cachePolicy: %s, total: %d, took: %s, cost: %d μs/op}\n",
cachePolicy, count, cost, costs/count/1000)
}
func createSearcher(dbPath string, cachePolicy string) (*xdb.Searcher, error) {
handle, err := os.OpenFile(dbPath, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open xdb file `%s`: %w", dbPath, err)
}
defer handle.Close()
// verify the xdb file
// @Note: do NOT call it every time you create a searcher since this will slow down the search response.
// @see the util.Verify function for details.
err = xdb.Verify(handle)
if err != nil {
return nil, fmt.Errorf("xdb verify: %w", err)
}
// auto-detect the ip version from the xdb header
header, err := xdb.LoadHeader(handle)
if err != nil {
return nil, fmt.Errorf("failed to load header from `%s`: %s", dbPath, err)
}
version, err := xdb.VersionFromHeader(header)
if err != nil {
return nil, fmt.Errorf("failed to detect IP version from `%s`: %s", dbPath, err)
}
switch cachePolicy {
case "nil", "file":
return xdb.NewWithFileOnly(version, dbPath)
case "vectorIndex":
vIndex, err := xdb.LoadVectorIndexFromFile(dbPath)
if err != nil {
return nil, fmt.Errorf("failed to load vector index from `%s`: %w", dbPath, err)
}
return xdb.NewWithVectorIndex(version, dbPath, vIndex)
case "content":
cBuff, err := xdb.LoadContentFromFile(dbPath)
if err != nil {
return nil, fmt.Errorf("failed to load content from '%s': %w", dbPath, err)
}
return xdb.NewWithBuffer(version, cBuff)
default:
return nil, fmt.Errorf("invalid cache policy `%s`, options: file/vectorIndex/content", cachePolicy)
}
}
func main() {
if len(os.Args) < 2 {
printHelp()
return
}
// set the log flag
log.SetFlags(log.Ldate | log.Ltime | log.Lshortfile)
switch strings.ToLower(os.Args[1]) {
case "search":
testSearch()
case "bench":
testBench()
default:
printHelp()
}
}

267
binding/golang/xdb/searcher.go
View File

@ -1,267 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// Ip2Region database v2.0 searcher.
// @Note this is a Not thread safe implementation.
//
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package xdb
import (
"encoding/binary"
"fmt"
"os"
)
const (
Structure20 = 2
Structure30 = 3
HeaderInfoLength = 256
VectorIndexRows = 256
VectorIndexCols = 256
VectorIndexSize = 8
)
// --- Index policy define
type IndexPolicy int
const (
VectorIndexPolicy IndexPolicy = 1
BTreeIndexPolicy IndexPolicy = 2
)
func (i IndexPolicy) String() string {
switch i {
case VectorIndexPolicy:
return "VectorIndex"
case BTreeIndexPolicy:
return "BtreeIndex"
default:
return "unknown"
}
}
// --- Header define
type Header struct {
// data []byte
Version uint16
IndexPolicy IndexPolicy
CreatedAt uint32
StartIndexPtr uint32
EndIndexPtr uint32
// since IPv6 supporting
IPVersion int
RuntimePtrBytes int
}
func NewHeader(input []byte) (*Header, error) {
if len(input) < 16 {
return nil, fmt.Errorf("invalid input buffer")
}
return &Header{
Version: binary.LittleEndian.Uint16(input[0:]),
IndexPolicy: IndexPolicy(binary.LittleEndian.Uint16(input[2:])),
CreatedAt: binary.LittleEndian.Uint32(input[4:]),
StartIndexPtr: binary.LittleEndian.Uint32(input[8:]),
EndIndexPtr: binary.LittleEndian.Uint32(input[12:]),
IPVersion: int(binary.LittleEndian.Uint16(input[16:])),
RuntimePtrBytes: int(binary.LittleEndian.Uint16(input[18:])),
}, nil
}
// --- searcher implementation
type Searcher struct {
version *Version
handle *os.File
ioCount int
// use it only when this feature enabled.
// Preload the vector index will reduce the number of IO operations
// thus speedup the search process
vectorIndex []byte
// content buffer.
// running with the whole xdb file cached
contentBuff []byte
}
func baseNew(version *Version, dbFile string, vIndex []byte, cBuff []byte) (*Searcher, error) {
var err error
// content buff first
if cBuff != nil {
return &Searcher{
version: version,
vectorIndex: nil,
contentBuff: cBuff,
}, nil
}
// open the xdb binary file
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, err
}
return &Searcher{
version: version,
handle: handle,
vectorIndex: vIndex,
}, nil
}
func NewWithFileOnly(version *Version, dbFile string) (*Searcher, error) {
return baseNew(version, dbFile, nil, nil)
}
func NewWithVectorIndex(version *Version, dbFile string, vIndex []byte) (*Searcher, error) {
return baseNew(version, dbFile, vIndex, nil)
}
func NewWithBuffer(version *Version, cBuff []byte) (*Searcher, error) {
return baseNew(version, "", nil, cBuff)
}
func (s *Searcher) Close() {
if s.handle != nil {
err := s.handle.Close()
if err != nil {
return
}
}
}
// IPVersion return the ip version
func (s *Searcher) IPVersion() *Version {
return s.version
}
// GetIOCount return the global io count for the last search
func (s *Searcher) GetIOCount() int {
return s.ioCount
}
// SearchByStr find the region for the specified ip string
func (s *Searcher) SearchByStr(str string) (string, error) {
ip, err := ParseIP(str)
if err != nil {
return "", err
}
return s.Search(ip)
}
// Search find the region for the specified long ip
func (s *Searcher) Search(ip []byte) (string, error) {
// ip version check
if len(ip) != s.version.Bytes {
return "", fmt.Errorf("invalid ip address(%s expected)", s.version.Name)
}
// reset the global ioCount
s.ioCount = 0
// locate the segment index block based on the vector index
var il0, il1 = int(ip[0]), int(ip[1])
var idx = il0*VectorIndexCols*VectorIndexSize + il1*VectorIndexSize
var sPtr, ePtr = uint32(0), uint32(0)
if s.vectorIndex != nil {
sPtr = binary.LittleEndian.Uint32(s.vectorIndex[idx:])
ePtr = binary.LittleEndian.Uint32(s.vectorIndex[idx+4:])
} else if s.contentBuff != nil {
sPtr = binary.LittleEndian.Uint32(s.contentBuff[HeaderInfoLength+idx:])
ePtr = binary.LittleEndian.Uint32(s.contentBuff[HeaderInfoLength+idx+4:])
} else {
// read the vector index block
var buff = make([]byte, VectorIndexSize)
err := s.read(int64(HeaderInfoLength+idx), buff)
if err != nil {
return "", fmt.Errorf("read vector index block at %d: %w", HeaderInfoLength+idx, err)
}
sPtr = binary.LittleEndian.Uint32(buff)
ePtr = binary.LittleEndian.Uint32(buff[4:])
}
// fmt.Printf("sPtr=%d, ePtr=%d\n", sPtr, ePtr)
// binary search the segment index to get the region
var bytes, dBytes = len(ip), len(ip) << 1
var segIndexSize = uint32(s.version.SegmentIndexSize)
var dataLen, dataPtr = 0, uint32(0)
var buff = make([]byte, segIndexSize)
var l, h = 0, int((ePtr - sPtr) / segIndexSize)
for l <= h {
m := (l + h) >> 1
p := sPtr + uint32(m)*segIndexSize
err := s.read(int64(p), buff)
if err != nil {
return "", fmt.Errorf("read segment index at %d: %w", p, err)
}
// decode the data step by step to reduce the unnecessary operations
if s.version.IPCompare(ip, buff[0:bytes]) < 0 {
h = m - 1
} else if s.version.IPCompare(ip, buff[bytes:dBytes]) > 0 {
l = m + 1
} else {
dataLen = int(binary.LittleEndian.Uint16(buff[dBytes:]))
dataPtr = binary.LittleEndian.Uint32(buff[dBytes+2:])
break
}
}
// fmt.Printf("dataLen: %d, dataPtr: %d\n", dataLen, dataPtr)
if dataLen == 0 {
return "", nil
}
// load and return the region data
var regionBuff = make([]byte, dataLen)
err := s.read(int64(dataPtr), regionBuff)
if err != nil {
return "", fmt.Errorf("read region at %d: %w", dataPtr, err)
}
return string(regionBuff), nil
}
// do the data read operation based on the setting.
// content buffer first or will read from the file.
// this operation will invoke the Seek for file based read.
func (s *Searcher) read(offset int64, buff []byte) error {
if s.contentBuff != nil {
cLen := copy(buff, s.contentBuff[offset:])
if cLen != len(buff) {
return fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
} else {
_, err := s.handle.Seek(offset, 0)
if err != nil {
return fmt.Errorf("seek to %d: %w", offset, err)
}
s.ioCount++
rLen, err := s.handle.Read(buff)
if err != nil {
return fmt.Errorf("handle read: %w", err)
}
if rLen != len(buff) {
return fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
}
return nil
}

245
binding/golang/xdb/util.go
View File

@ -1,245 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package xdb
import (
"bytes"
"embed"
"fmt"
"io"
"net"
"os"
)
func ParseIP(ip string) ([]byte, error) {
parsedIP := net.ParseIP(ip)
if parsedIP == nil {
return nil, fmt.Errorf("invalid ip address: %s", ip)
}
v4 := parsedIP.To4()
if v4 != nil {
return v4, nil
}
v6 := parsedIP.To16()
if v6 != nil {
return v6, nil
}
return nil, fmt.Errorf("invalid ip address: %s", ip)
}
func IP2String(ip []byte) string {
return net.IP(ip[:]).String()
}
// IPCompare compares two IP addresses
// Returns: -1 if ip1 < ip2, 0 if ip1 == ip2, 1 if ip1 > ip2
func IPCompare(ip1, ip2 []byte) int {
// for i := 0; i < len(ip1); i++ {
// if ip1[i] < ip2[i] {
// return -1
// }
// if ip1[i] > ip2[i] {
// return 1
// }
// }
// return 0
return bytes.Compare(ip1, ip2)
}
// Verify if the current Searcher could be used to search the specified xdb file.
// Why do we need this check ?
// The future features of the xdb impl may cause the current searcher not able to work properly.
//
// @Note: You Just need to check this ONCE when the service starts
// Or use another process (eg, A command) to check once Just to confirm the suitability.
func Verify(handle *os.File) error {
header, err := LoadHeader(handle)
if err != nil {
return fmt.Errorf("loading header: %w", err)
}
// get the runtime ptr bytes
runtimePtrBytes := 0
switch header.Version {
case Structure20:
runtimePtrBytes = 4
case Structure30:
runtimePtrBytes = header.RuntimePtrBytes
default:
return fmt.Errorf("invalid version: %d", header.Version)
}
// 1, confirm the xdb file size.
// to sure that the MaxFilePointer does no overflow
stat, err := handle.Stat()
if err != nil {
return fmt.Errorf("file stat: %w", err)
}
maxFilePtr := int64(1<<(runtimePtrBytes*8) - 1)
if stat.Size() > maxFilePtr {
return fmt.Errorf("xdb file exceeds the maximum supported bytes: %d", maxFilePtr)
}
return nil
}
// VerifyFromFile check Verify for details
func VerifyFromFile(dbFile string) error {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return fmt.Errorf("open xdb file `%s`: %w", dbFile, err)
}
defer handle.Close()
return Verify(handle)
}
// LoadHeader load the header info from the specified handle
func LoadHeader(handle *os.File) (*Header, error) {
_, err := handle.Seek(0, 0)
if err != nil {
return nil, fmt.Errorf("seek to the header: %w", err)
}
var buff = make([]byte, HeaderInfoLength)
rLen, err := handle.Read(buff)
if err != nil {
return nil, err
}
if rLen != len(buff) {
return nil, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
return NewHeader(buff)
}
// LoadHeaderFromFile load header info from the specified db file path
func LoadHeaderFromFile(dbFile string) (*Header, error) {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open xdb file `%s`: %w", dbFile, err)
}
defer handle.Close()
header, err := LoadHeader(handle)
if err != nil {
return nil, err
}
return header, nil
}
// LoadHeaderFromBuff wrap the header info from the content buffer
func LoadHeaderFromBuff(cBuff []byte) (*Header, error) {
return NewHeader(cBuff[0:HeaderInfoLength])
}
// LoadVectorIndex util function to load the vector index from the specified file handle
func LoadVectorIndex(handle *os.File) ([]byte, error) {
// load all the vector index block
_, err := handle.Seek(HeaderInfoLength, 0)
if err != nil {
return nil, fmt.Errorf("seek to vector index: %w", err)
}
var buff = make([]byte, VectorIndexRows*VectorIndexCols*VectorIndexSize)
rLen, err := handle.Read(buff)
if err != nil {
return nil, err
}
if rLen != len(buff) {
return nil, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
return buff, nil
}
// LoadVectorIndexFromFile load vector index from a specified file path
func LoadVectorIndexFromFile(dbFile string) ([]byte, error) {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open xdb file `%s`: %w", dbFile, err)
}
defer handle.Close()
vIndex, err := LoadVectorIndex(handle)
if err != nil {
return nil, err
}
return vIndex, nil
}
// LoadContent load the whole xdb content from the specified file handle
func LoadContent(handle *os.File) ([]byte, error) {
// get file size
fi, err := handle.Stat()
if err != nil {
return nil, fmt.Errorf("stat: %w", err)
}
size := fi.Size()
// seek to the head of the file
_, err = handle.Seek(0, 0)
if err != nil {
return nil, fmt.Errorf("seek to get xdb file length: %w", err)
}
var buff = make([]byte, size)
rLen, err := handle.Read(buff)
if err != nil {
return nil, err
}
if rLen != len(buff) {
return nil, fmt.Errorf("incomplete read: readed bytes should be %d", len(buff))
}
return buff, nil
}
// LoadContentFromFile load the whole xdb content from the specified db file path
func LoadContentFromFile(dbFile string) ([]byte, error) {
handle, err := os.OpenFile(dbFile, os.O_RDONLY, 0600)
if err != nil {
return nil, fmt.Errorf("open xdb file `%s`: %w", dbFile, err)
}
defer handle.Close()
cBuff, err := LoadContent(handle)
if err != nil {
return nil, err
}
return cBuff, nil
}
// LoadContentFromFS load the whole xdb binary from embed.FS
func LoadContentFromFS(fs embed.FS, filePath string) ([]byte, error) {
file, err := fs.Open(filePath)
if err != nil {
return nil, fmt.Errorf("failed to open embedded file `%s`: %w", filePath, err)
}
defer file.Close()
var cBuff []byte
cBuff, err = io.ReadAll(file)
if err != nil {
return nil, fmt.Errorf("failed to read embedded file `%s`: %w", filePath, err)
}
return cBuff, nil
}

78
binding/golang/xdb/util_test.go
View File

@ -1,78 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/16
package xdb
import (
"fmt"
"testing"
"time"
)
func TestParseIP(t *testing.T) {
var ips = []string{"29.34.191.255", "2c0f:fff0::", "2fff:ffff:ffff:ffff:ffff:ffff:ffff:ffff"}
for _, ip := range ips {
bytes, err := ParseIP(ip)
if err != nil {
t.Errorf("check ip `%s`: %s\n", IP2String(bytes), err)
}
nip := IP2String(bytes)
fmt.Printf("checkip: (%s / %s), isEqual: %v\n", ip, nip, ip == nip)
}
}
func TestIPCompare(t *testing.T) {
var ipPairs = [][]string{
{"1.2.3.4", "1.2.3.5"},
{"58.250.36.41", "58.250.30.41"},
{"2c10::", "2e00::"},
{"fdff:ffff:ffff:ffff:ffff:ffff:ffff:ffff", "febf:ffff:ffff:ffff:ffff:ffff:ffff:ffff"},
{"fe7f:ffff:ffff:ffff:ffff:ffff:ffff:ffff", "fe00::"},
}
for _, pairs := range ipPairs {
fmt.Printf("IPCompare(%s, %s): %d\n", pairs[0], pairs[1], IPCompare([]byte(pairs[0]), []byte(pairs[1])))
}
}
func TestLoadVectorIndex(t *testing.T) {
vIndex, err := LoadVectorIndexFromFile("../../../data/ip2region_v4.xdb")
if err != nil {
fmt.Printf("failed to load vector index: %s\n", err)
return
}
fmt.Printf("vIndex length: %d\n", len(vIndex))
}
func TestLoadContent(t *testing.T) {
buff, err := LoadContentFromFile("../../../data/ip2region_v4.xdb")
if err != nil {
fmt.Printf("failed to load xdb content: %s\n", err)
return
}
fmt.Printf("buff length: %d\n", len(buff))
}
func TestLoadHeader(t *testing.T) {
header, err := LoadHeaderFromFile("../../../data/ip2region_v4.xdb")
if err != nil {
fmt.Printf("failed to load xdb header info: %s\n", err)
return
}
fmt.Printf("Version : %d\n", header.Version)
fmt.Printf("IndexPolicy : %s\n", header.IndexPolicy.String())
fmt.Printf("CreatedAt : %d(%s)\n", header.CreatedAt, time.Unix(int64(header.CreatedAt), 0).Format(time.RFC3339))
fmt.Printf("StartIndexPtr : %d\n", header.StartIndexPtr)
fmt.Printf("EndIndexPtr : %d\n", header.EndIndexPtr)
fmt.Printf("IPVersion : %d\n", header.IPVersion)
fmt.Printf("RuntimePtrBytes : %d\n", header.RuntimePtrBytes)
}

97
binding/golang/xdb/version.go
View File

@ -1,97 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package xdb
import (
"bytes"
"fmt"
"strings"
)
type Version struct {
Id int
Name string
Bytes int
SegmentIndexSize int
// function to compare two ips
IPCompare func([]byte, []byte) int
}
const (
IPv4VersionNo = 4
IPv6VersionNo = 6
)
var (
IPvx = &Version{}
IPv4 = &Version{
Id: IPv4VersionNo,
Name: "IPv4",
Bytes: 4,
SegmentIndexSize: 14, // 4 + 4 + 2 + 4,
IPCompare: func(ip1, ip2 []byte) int {
// ip1 - with Big endian byte order parsed from an input
// ip2 - with Little endian byte order read from the xdb index
ip2[0], ip2[3] = ip2[3], ip2[0]
ip2[1], ip2[2] = ip2[2], ip2[1]
return bytes.Compare(ip1, ip2)
},
}
IPv6 = &Version{
Id: IPv6VersionNo,
Name: "IPv6",
Bytes: 16,
SegmentIndexSize: 38, // 16 + 16 + 2 + 4,
IPCompare: func(ip1, ip2 []byte) int {
return bytes.Compare(ip1, ip2)
},
}
)
func VersionFromIP(ip string) (*Version, error) {
r, err := ParseIP(ip)
if err != nil {
return IPvx, fmt.Errorf("parse ip fail: %w", err)
}
if len(r) == 4 {
return IPv4, nil
}
return IPv6, nil
}
func VersionFromName(name string) (*Version, error) {
switch strings.ToUpper(name) {
case "V4", "IPV4":
return IPv4, nil
case "V6", "IPV6":
return IPv6, nil
default:
return IPvx, fmt.Errorf("invalid version name `%s`", name)
}
}
func VersionFromHeader(header *Header) (*Version, error) {
// old structure with IPv4 supports ONLY
if header.Version == Structure20 {
return IPv4, nil
}
// structure 3.0 after IPv6 supporting
if header.Version != Structure30 {
return IPvx, fmt.Errorf("invalid version `%d`", header.IPVersion)
}
switch header.IPVersion {
case IPv4VersionNo:
return IPv4, nil
case IPv6VersionNo:
return IPv6, nil
default:
return IPvx, fmt.Errorf("invalid version `%d`", header.Version)
}
}

256
binding/java/ReadMe.md
View File

@ -1,256 +0,0 @@
# ip2region xdb java 查询客户端实现
# 使用方式
### maven 仓库：
```xml
<dependency>
<groupId>org.lionsoul</groupId>
<artifactId>ip2region</artifactId>
<version>3.1.0</version>
</dependency>
```
### 关于 IPv4 和 IPv6
该 xdb 查询客户端实现同时支持对 IPv4 和 IPv6 的查询，使用方式如下：
```java
import org.lionsoul.ip2region.xdb.Version;
// 如果是 IPv4: 设置 xdb 路径为 v4 的 xdb 文件，IP版本指定为 Version.IPv4
final String dbPath = "../../data/ip2region_v4.xdb"; // 或者你的 ipv4 xdb 的路径
final Version version = Version.IPv4;
// 如果是 IPv6: 设置 xdb 路径为 v6 的 xdb 文件，IP版本指定为 Version.IPv6
final String dbPath = "../../data/ip2region_v6.xdb"; // 或者你的 ipv6 xdb 路径
final Version version = Version.IPv6;
// dbPath 指定的 xdb 的 IP 版本必须和 version 指定的一致，不然查询执行的时候会报错
// 备注：以下演示直接使用 dbPath 和 version 变量
```
### 文件验证
建议您主动去验证 xdb 文件的适用性，因为后期的一些新功能可能会导致目前的 Searcher 版本无法适用你使用的 xdb 文件，验证可以避免运行过程中的一些不可预测的错误。 你不需要每次都去验证，例如在服务启动的时候，或者手动调用命令验证确认版本匹配即可，不要在每次创建的 Searcher 的时候运行验证，这样会影响查询的响应速度，尤其是高并发的使用场景。
```java
try {
Searcher.verifyFromFile(dbPath);
} catch (Exception e) {
// 适用性验证失败！！！
// 当前查询客户端实现不适用于 dbPath 指定的 xdb 文件的查询.
// 应该停止启动服务，使用合适的 xdb 文件或者升级到适合 dbPath 的 Searcher 实现。
return;
}
// 验证通过，当前使用的 Searcher 可以安全的用于对 dbPath 指向的 xdb 的查询操作
```
### 完全基于文件的查询
```java
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearcherTest {
public static void main(String[] args) {
// 1、使用上述的 version 和 dbPath 创建 searcher 对象
Searcher searcher = null;
try {
searcher = Searcher.newWithFileOnly(version, dbPath);
} catch (IOException e) {
System.out.printf("failed to create searcher with `%s`: %s\n", dbPath, e);
return;
}
// 2、查询，IPv4 或者 IPv6 的地址都支持
try {
String ip = "1.2.3.4";
// ip = "2001:4:112:ffff:ffff:ffff:ffff:ffff"; // IPv6
long sTime = System.nanoTime();
String region = searcher.search(ip);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("failed to search(%s): %s\n", ip, e);
}
// 3、关闭资源
searcher.close();
// 备注：并发使用，每个线程需要创建一个独立的 searcher 对象单独使用。
}
}
```
### 缓存 `VectorIndex` 索引
我们可以提前从 `xdb` 文件中加载出来 `VectorIndex` 数据，然后全局缓存，每次创建 Searcher 对象的时候使用全局的 VectorIndex 缓存可以减少一次固定的 IO 操作，从而加速查询，减少 IO 压力。
```java
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearcherTest {
public static void main(String[] args) {
// 备注：version 和 dbPath 来源，请看上面的版本描述
// 1、从 dbPath 中预先加载 VectorIndex 缓存，并且把这个得到的数据作为全局变量，后续反复使用。
byte[] vIndex;
try {
vIndex = Searcher.loadVectorIndexFromFile(dbPath);
} catch (Exception e) {
System.out.printf("failed to load vector index from `%s`: %s\n", dbPath, e);
return;
}
// 2、使用全局的 vIndex 创建带 VectorIndex 缓存的查询对象。
Searcher searcher;
try {
searcher = Searcher.newWithVectorIndex(version, dbPath, vIndex);
} catch (Exception e) {
System.out.printf("failed to create vectorIndex cached searcher with `%s`: %s\n", dbPath, e);
return;
}
// 3、查询，IPv4 或者 IPv6 地址都支持
try {
String ip = "1.2.3.4";
// ip = "2001:4:112:ffff:ffff:ffff:ffff:ffff"; // IPv6
long sTime = System.nanoTime();
String region = searcher.search(ip);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("failed to search(%s): %s\n", ip, e);
}
// 4、关闭资源
searcher.close();
// 备注：每个线程需要单独创建一个独立的 Searcher 对象，但是都共享全局的制度 vIndex 缓存。
}
}
```
### 缓存整个 `xdb` 数据
我们也可以预先加载整个 ip2region.xdb 的数据到内存，然后基于这个数据创建查询对象来实现完全基于文件的查询，类似之前的 memory search。
```java
import org.lionsoul.ip2region.xdb.Searcher;
import java.io.*;
import java.util.concurrent.TimeUnit;
public class SearcherTest {
public static void main(String[] args) {
// 备注：version 和 dbPath 来源，请看上面的版本描述
// 1、从 dbPath 加载整个 xdb 到内存。
// 从这个 release 版本开始，xdb 的 buffer 使用 LongByteArray 来存储，避免 xdb 文件过大的时候 int 类型的溢出
LongByteArray cBuff;
try {
cBuff = Searcher.loadContentFromFile(dbPath);
} catch (Exception e) {
System.out.printf("failed to load content from `%s`: %s\n", dbPath, e);
return;
}
// 2、使用上述的 cBuff 创建一个完全基于内存的查询对象。
Searcher searcher;
try {
searcher = Searcher.newWithBuffer(version, cBuff);
} catch (Exception e) {
System.out.printf("failed to create content cached searcher: %s\n", e);
return;
}
// 3、查询，IPv4 和 IPv6 都支持
try {
String ip = "1.2.3.4";
// ip = "2001:4:112:ffff:ffff:ffff:ffff:ffff"; // IPv6
long sTime = System.nanoTime();
String region = searcher.search(ip);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("failed to search(%s): %s\n", ip, e);
}
// 4、关闭资源 - 该 searcher 对象可以安全用于并发，等整个服务关闭的时候再关闭 searcher
// searcher.close();
// 备注：并发使用，用整个 xdb 数据缓存创建的查询对象可以安全的用于并发，也就是你可以把这个 searcher 对象做成全局对象去跨线程访问。
}
}
```
# 编译测试程序
通过 maven 来编译测试程序。
```bash
# cd 到 java binding 的根目录
cd binding/java/
mvn compile package
```
然后会在当前目录的 target 目录下得到一个 ip2region-{version}.jar 的打包文件。
# 查询测试
可以通过 `java -jar ip2region-{version}.jar search` 命令来测试查询：
```bash
➜ java git:(fr_java_ipv6) ✗ java -jar target/ip2region-3.1.0.jar search
java -jar ip2region-{version}.jar search [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：使用默认的 data/ip2region_v4.xdb 文件进行 IPv4 的查询测试：
```bash
➜ java git:(fr_java_ipv6) ✗ java -jar target/ip2region-3.1.0.jar search --db=../../data/ip2region_v4.xdb
ip2region xdb searcher test program
source xdb: ../../data/ip2region_v4.xdb (IPv4, vectorIndex)
type 'quit' to exit
ip2region>> 1.2.3.4
{region: 美国|华盛顿|0|谷歌, ioCount: 7, took: 82 μs}
```
例如：使用默认的 data/ip2region_v6.xdb 文件进行 IPv6 的查询测试：
```bash
➜ java git:(fr_java_ipv6) ✗ java -jar target/ip2region-3.1.0.jar search --db=../../data/ip2region_v6.xdb
ip2region xdb searcher test program
source xdb: ../../data/ip2region_v6.xdb (IPv6, vectorIndex)
type 'quit' to exit
ip2region>> 240e:3b7:3272:d8d0:db09:c067:8d59:539e
{region: 中国|广东省|深圳市|家庭宽带, ioCount: 14, took: 424 μs}
```
输入 ip 即可进行查询测试，也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的查询效果。
# bench 测试
可以通过 `java -jar ip2region-{version}.jar bench` 命令来进行 bench 测试，一方面确保 `xdb` 文件没有错误，一方面可以评估查询性能：
```bash
➜ java git:(fr_java_ipv6) ✗ java -jar target/ip2region-3.1.0.jar bench
java -jar ip2region-{version}.jar bench [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：通过默认的 data/ip2region_v4.xdb 和 data/ipv4_source.txt 文件进行 IPv4 的 bench 测试：
```bash
java -jar target/ip2region-3.1.0.jar bench --db=../../data/ip2region_v4.xdb --src=../../data/ipv4_source.txt
```
例如：通过默认的 data/ip2region_v6.xdb 和 data/ipv6_source.txt 文件进行 IPv6 的 bench 测试：
```bash
java -jar target/ip2region-3.1.0.jar bench --db=../../data/ip2region_v6.xdb --src=../../data/ipv6_source.txt
```
可以通过分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的效果。
@Note: 注意 bench 使用的 src 文件要是生成对应 xdb 文件相同的源文件。

204
binding/java/pom.xml
View File

@ -1,204 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.lionsoul</groupId>
<artifactId>ip2region</artifactId>
<version>3.1.0</version>
<packaging>jar</packaging>
<name>ip2region</name>
<url>https://github.com/lionsoul2014/ip2region</url>
<description>An open source offline IP address data manager framework and locator with both IPv4 and IPv6 suppported</description>
<licenses>
<license>
<name>The Apache Software License, Version 2.0</name>
<url>https://www.apache.org/licenses/LICENSE-2.0.txt</url>
<distribution>repo</distribution>
</license>
</licenses>
<scm>
<url>git@github.com:lionsoul2014/ip2region.git</url>
<connection>scm:git:git@github.com:lionsoul2014/ip2region.git</connection>
<developerConnection>scm:git:git@github.com:lionsoul2014/ip2region.git</developerConnection>
</scm>
<developers>
<developer>
<id>lionsoul</id>
<name>chenxin</name>
<email>chenxin619315@gmail.com</email>
</developer>
</developers>
<distributionManagement>
<snapshotRepository>
<id>lionsoul</id>
<url>https://oss.sonatype.org/content/repositories/snapshots/</url>
</snapshotRepository>
<repository>
<id>lionsoul</id>
<url>https://oss.sonatype.org/service/local/staging/deploy/maven2/</url>
</repository>
</distributionManagement>
<issueManagement>
<url>https://github.com/lionsoul2014/ip2region/issues</url>
<system>Github issues</system>
</issueManagement>
<properties>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<maven.compiler.source>1.8</maven.compiler.source>
<maven.compiler.target>1.8</maven.compiler.target>
</properties>
<dependencies>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<version>4.13.1</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.1.2</version>
<executions>
<execution>
<id>attach-sources</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9</version>
<executions>
<execution>
<id>attach-javadocs</id>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
<configuration>
<additionalparam>${javadoc.opts}</additionalparam>
</configuration>
</execution>
</executions>
<configuration>
<failOnError>false</failOnError>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>1.4</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<transformers>
<transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
<mainClass>org.lionsoul.ip2region.SearchApp</mainClass>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
<profiles>
<profile>
<id>java8-doclint-disabled</id>
<activation>
<jdk>[1.8,)</jdk>
</activation>
<properties>
<javadoc.opts>-Xdoclint:none</javadoc.opts>
</properties>
</profile>
<profile>
<id>release</id>
<build>
<plugins>
<!-- Source -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
<version>2.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>jar-no-fork</goal>
</goals>
</execution>
</executions>
</plugin>
<!-- Javadoc -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-javadoc-plugin</artifactId>
<version>2.9.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>jar</goal>
</goals>
<configuration>
<additionalparam>${javadoc.opts}</additionalparam>
</configuration>
</execution>
</executions>
</plugin>
<!-- GPG -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-gpg-plugin</artifactId>
<version>1.5</version>
<executions>
<execution>
<phase>verify</phase>
<goals>
<goal>sign</goal>
</goals>
</execution>
</executions>
</plugin>
<!--use the Central Portal: https://central.sonatype.org/publish/publish-portal-maven/ instead-->
<plugin>
<groupId>org.sonatype.central</groupId>
<artifactId>central-publishing-maven-plugin</artifactId>
<version>0.7.0</version>
<extensions>true</extensions>
<configuration>
<publishingServerId>lionsoul</publishingServerId>
<autoPublish>true</autoPublish>
</configuration>
</plugin>
</plugins>
</build>
</profile>
</profiles>
</project>

251
binding/java/src/main/java/org/lionsoul/ip2region/SearchApp.java
View File

@ -1,251 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
package org.lionsoul.ip2region;
import org.lionsoul.ip2region.xdb.InetAddressException;
import org.lionsoul.ip2region.xdb.XdbException;
import org.lionsoul.ip2region.xdb.LongByteArray;
import org.lionsoul.ip2region.xdb.Searcher;
import org.lionsoul.ip2region.xdb.Util;
import org.lionsoul.ip2region.xdb.Version;
import java.io.*;
import java.nio.charset.Charset;
import java.util.concurrent.TimeUnit;
public class SearchApp {
public static void printHelp(String[] args) {
System.out.print("ip2region xdb searcher\n");
System.out.print("java -jar ip2region-{version}.jar [command] [command options]\n");
System.out.print("Command: \n");
System.out.print(" search search input test\n");
System.out.print(" bench search bench test\n");
}
public static Searcher createSearcher(String dbPath, String cachePolicy) throws IOException, XdbException {
final RandomAccessFile handle = new RandomAccessFile(dbPath, "r");
// verify the xdb file
// @Note: do NOT call it every time you create a searcher since this will slow
// down the search response.
// @see the util.Verify function for details.
Searcher.verify(handle);
// get the ip version from header
final Version version = Version.fromHeader(Searcher.loadHeader(handle));
// create the final searcher
if ("file".equals(cachePolicy)) {
return Searcher.newWithFileOnly(version, dbPath);
} else if ("vectorIndex".equals(cachePolicy)) {
byte[] vIndex = Searcher.loadVectorIndexFromFile(dbPath);
return Searcher.newWithVectorIndex(version, dbPath, vIndex);
} else if ("content".equals(cachePolicy)) {
LongByteArray cBuff = Searcher.loadContentFromFile(dbPath);
return Searcher.newWithBuffer(version, cBuff);
} else {
throw new IOException("invalid cache policy `" + cachePolicy + "`, options: file/vectorIndex/content");
}
}
public static void searchTest(String[] args) throws IOException, XdbException {
String dbPath = "", cachePolicy = "vectorIndex";
for (final String r : args) {
if (r.length() < 5) {
continue;
}
if (r.indexOf("--") != 0) {
continue;
}
int sIdx = r.indexOf('=');
if (sIdx < 0) {
System.out.printf("missing = for args pair `%s`\n", r);
return;
}
String key = r.substring(2, sIdx);
String val = r.substring(sIdx + 1);
// System.out.printf("key=%s, val=%s\n", key, val);
if ("db".equals(key)) {
dbPath = val;
} else if ("cache-policy".equals(key)) {
cachePolicy = val;
} else {
System.out.printf("undefined option `%s`\n", r);
return;
}
}
if (dbPath.isEmpty()) {
System.out.print("java -jar ip2region-{version}.jar search [command options]\n");
System.out.print("options:\n");
System.out.print(" --db string ip2region binary xdb file path\n");
System.out.print(" --cache-policy string cache policy: file/vectorIndex/content\n");
return;
}
Searcher searcher = createSearcher(dbPath, cachePolicy);
final BufferedReader reader = new BufferedReader(new InputStreamReader(System.in));
System.out.printf("ip2region xdb searcher test program\n"
+ "source xdb: %s (%s, %s)\n"
+ "type 'quit' to exit\n", dbPath, searcher.getIPVersion().name, cachePolicy);
while ( true ) {
System.out.print("ip2region>> ");
String line = reader.readLine().trim();
if ( line.length() < 2 ) {
continue;
}
if ( line.equalsIgnoreCase("quit") ) {
break;
}
try {
double sTime = System.nanoTime();
String region = searcher.search(line);
long cost = TimeUnit.NANOSECONDS.toMicros((long) (System.nanoTime() - sTime));
System.out.printf("{region: %s, ioCount: %d, took: %d μs}\n", region, searcher.getIOCount(), cost);
} catch (Exception e) {
System.out.printf("{err: %s, ioCount: %d}\n", e, searcher.getIOCount());
}
}
reader.close();
searcher.close();
System.out.println("searcher test program exited, thanks for trying");
}
public static void benchTest(String[] args) throws IOException, XdbException, InetAddressException {
String dbPath = "", srcPath = "", cachePolicy = "vectorIndex";
for (final String r : args) {
if (r.length() < 5) {
continue;
}
if (r.indexOf("--") != 0) {
continue;
}
int sIdx = r.indexOf('=');
if (sIdx < 0) {
System.out.printf("missing = for args pair `%s`\n", r);
return;
}
String key = r.substring(2, sIdx);
String val = r.substring(sIdx + 1);
if ("db".equals(key)) {
dbPath = val;
} else if ("src".equals(key)) {
srcPath = val;
} else if ("cache-policy".equals(key)) {
cachePolicy = val;
} else {
System.out.printf("undefined option `%s`\n", r);
return;
}
}
if (dbPath.length() < 1 || srcPath.length() < 1) {
System.out.print("java -jar ip2region-{version}.jar bench [command options]\n");
System.out.print("options:\n");
System.out.print(" --db string ip2region binary xdb file path\n");
System.out.print(" --src string source ip text file path\n");
System.out.print(" --cache-policy string cache policy: file/vectorIndex/content\n");
return;
}
Searcher searcher = createSearcher(dbPath, cachePolicy);
long count = 0, costs = 0, tStart = System.nanoTime();
String line;
final Charset charset = Charset.forName("utf-8");
final FileInputStream fis = new FileInputStream(srcPath);
final BufferedReader reader = new BufferedReader(new InputStreamReader(fis, charset));
while ((line = reader.readLine()) != null) {
String l = line.trim();
String[] ps = l.split("\\|", 3);
if (ps.length != 3) {
reader.close();
System.out.printf("invalid ip segment `%s`\n", l);
return;
}
byte[] sip;
try {
sip = Util.parseIP(ps[0]);
} catch (Exception e) {
reader.close();
System.out.printf("check start ip `%s`: %s\n", ps[0], e);
return;
}
byte[] eip;
try {
eip = Util.parseIP(ps[1]);
} catch (Exception e) {
reader.close();
System.out.printf("check end ip `%s`: %s\n", ps[1], e);
return;
}
if (Util.ipCompare(sip, eip) > 0) {
reader.close();
System.out.printf("start ip(%s) should not be greater than end ip(%s)\n", ps[0], ps[1]);
return;
}
for (final byte[] ip : new byte[][]{sip, eip}) {
long sTime = System.nanoTime();
String region = searcher.search(ip);
costs += System.nanoTime() - sTime;
// check the region info
if (!ps[2].equals(region)) {
System.out.printf("failed search(%s) with (%s != %s)\n", Util.ipToString(ip), region, ps[2]);
reader.close();
return;
}
count++;
}
}
reader.close();
searcher.close();
long took = System.nanoTime() - tStart;
System.out.printf("Bench finished, {cachePolicy: %s, total: %d, took: %ds, cost: %d μs/op}\n",
cachePolicy, count, TimeUnit.NANOSECONDS.toSeconds(took),
count == 0 ? 0 : TimeUnit.NANOSECONDS.toMicros(costs/count));
}
public static void main(String[] args) {
if (args.length < 1) {
printHelp(args);
return;
}
if ("search".equals(args[0])) {
try {
searchTest(args);
} catch (Exception e) {
System.out.printf("failed running search test: %s\n", e);
}
} else if ("bench".equals(args[0])) {
try {
benchTest(args);
} catch (Exception e) {
System.out.printf("fwailed running bench test: %s\n", e);
}
} else {
printHelp(args);
}
}
}

45
binding/java/src/main/java/org/lionsoul/ip2region/xdb/Header.java
View File

@ -1,45 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
package org.lionsoul.ip2region.xdb;
public class Header {
public final int version;
public final int indexPolicy;
public final long createdAt;
public final long startIndexPtr;
public final long endIndexPtr;
// since xdb 3.0 with IPv6 supporting
public final int ipVersion;
public final int runtimePtrBytes;
public final byte[] buffer;
public Header(byte[] buff) {
assert buff.length >= 16;
version = LittleEndian.getInt2(buff, 0);
indexPolicy = LittleEndian.getInt2(buff, 2);
createdAt = LittleEndian.getUint32(buff, 4);
startIndexPtr = LittleEndian.getUint32(buff, 8);
endIndexPtr = LittleEndian.getUint32(buff, 12);
ipVersion = LittleEndian.getInt2(buff, 16);
runtimePtrBytes = LittleEndian.getInt2(buff, 18);
buffer = buff;
}
@Override public String toString() {
return "{" +
"Version: " + version + ',' +
"IndexPolicy: " + indexPolicy + ',' +
"CreatedAt: " + createdAt + ',' +
"StartIndexPtr: " + startIndexPtr + ',' +
"EndIndexPtr: " + endIndexPtr + ',' +
"IPVersion: " + ipVersion + ',' +
"RuntimePtrBytes: " + runtimePtrBytes +
'}';
}
}

47
binding/java/src/main/java/org/lionsoul/ip2region/xdb/IPv4.java
View File

@ -1,47 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package org.lionsoul.ip2region.xdb;
// IPv4 version implementation
// @Author Lion <chenxin619315@gmail.com>
// @Date 2025/09/10
public class IPv4 extends Version {
public IPv4() {
// segmentIndex: 4 + 4 + 2 + 4
super(4, "IPv4", 4, 14);
}
@Override
public int putBytes(byte[] buff, int offset, byte[] ip) {
// use the Little endian byte order to compatible with the old searcher implementation
buff[offset++] = ip[3];
buff[offset++] = ip[2];
buff[offset++] = ip[1];
buff[offset ] = ip[0];
return ip.length;
}
@Override
public int ipSubCompare(byte[] ip1, byte[] buff, int offset) {
// ip1: Big endian byte order parsed from input
// ip2: Little endian byte order read from xdb index.
// @Note: to compatible with the old Litten endian index encode implementation.
int j = offset + ip1.length - 1;
for (int i = 0; i < ip1.length; i++, j--) {
final int i1 = (int) (ip1[i] & 0xFF);
final int i2 = (int) (buff[j] & 0xFF);
if (i1 < i2) {
return -1;
}
if (i1 > i2) {
return 1;
}
}
return 0;
}
}

29
binding/java/src/main/java/org/lionsoul/ip2region/xdb/IPv6.java
View File

@ -1,29 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package org.lionsoul.ip2region.xdb;
// IPv4 version implementation
// @Author Lion <chenxin619315@gmail.com>
// @Date 2025/09/10
public class IPv6 extends Version {
public IPv6() {
// segmentIndex: 16 + 16 + 2 + 4
super(6, "IPv6", 16, 38);
}
@Override
public int putBytes(byte[] buff, int offset, byte[] ip) {
System.arraycopy(ip, 0, buff, offset, ip.length);
return ip.length;
}
@Override
public int ipSubCompare(byte[] ip1, byte[] buff, int offset) {
return Util.ipSubCompare(ip1, buff, offset);
}
}

13
binding/java/src/main/java/org/lionsoul/ip2region/xdb/InetAddressException.java
View File

@ -1,13 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package org.lionsoul.ip2region.xdb;
public class InetAddressException extends Exception {
public InetAddressException(String str) {
super(str);
}
}

58
binding/java/src/main/java/org/lionsoul/ip2region/xdb/LittleEndian.java
View File

@ -1,58 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package org.lionsoul.ip2region.xdb;
// Little Endian basic data type decode and encode.
// @Author Lion <chenxin619315@gmail.com>
// @Date 2025/09/10
public class LittleEndian {
public final static int[] shiftIndex = {0, 8, 16, 24, 32, 40, 48, 56};
// put specified bytes to the buffer started from the offset
public static void put(final byte[] buff, int offset, long value, int bytes) {
if (bytes > 8) {
throw new IndexOutOfBoundsException("bytes should be <= 8");
}
for (int i = 0; i < bytes; i++) {
buff[offset++] = (byte)((value >>> shiftIndex[i]) & 0xFF);
}
}
// put an uint32 (4 bytes long) to the buffer from the offset
public static void putUint32(final byte[] buff, int offset, long value) {
buff[offset++] = (byte) (value & 0xFF);
buff[offset++] = (byte) ((value >> 8) & 0xFF);
buff[offset++] = (byte) ((value >> 16) & 0xFF);
buff[offset ] = (byte) ((value >> 24) & 0xFF);
}
// put a 2-bytes int to the buffer from the specified offset
public static void putInt2(final byte[] buff, int offset, int value) {
buff[offset++] = (byte) (value & 0xFF);
buff[offset ] = (byte) ((value >> 8) & 0xFF);
}
// get an uint32 from a byte array from the specified offset
public static long getUint32(final byte[] buff, int offset) {
return (
((buff[offset++] & 0x000000FFL)) |
((buff[offset++] << 8) & 0x0000FF00L) |
((buff[offset++] << 16) & 0x00FF0000L) |
((buff[offset ] << 24) & 0xFF000000L)
);
}
// get an 2 bytes int from a byte array from the specified offset
public static int getInt2(final byte[] buff, int offset) {
return (
((buff[offset++]) & 0x000000FF) |
((buff[offset ] << 8) & 0x0000FF00)
);
}
}

119
binding/java/src/main/java/org/lionsoul/ip2region/xdb/Log.java
View File

@ -1,119 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
//
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/07/14
package org.lionsoul.ip2region.xdb;
import java.text.SimpleDateFormat;
import java.util.Date;
// simple log implementation
public class Log {
/* Log level constants define */
public static final int DEBUG = 0;
public static final int INFO = 1;
public static final int WARN = 2;
public static final int ERROR = 3;
// level name
public static final String[] level_string = new String[] {
"DEBUG",
"INFO",
"WARN",
"ERROR"
};
public final Class<?> baseClass;
private int level = INFO;
public Log(Class<?> baseClass) {
this.baseClass = baseClass;
}
public static Log getLogger(Class<?> baseClass) {
return new Log(baseClass);
}
public String format(int level, String format, Object... args) {
// append the datetime
final StringBuilder sb = new StringBuilder();
final SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
sb.append(String.format("%s %-5s ", sdf.format(new Date()), level_string[level]));
// append the class name
sb.append(baseClass.getName()).append(' ');
sb.append(String.format(format, args));
return sb.toString();
}
public void printf(int level, String format, Object... args) {
if (level < DEBUG || level > ERROR) {
throw new IndexOutOfBoundsException("invalid level index " + level);
}
// level filter
if (level < this.level) {
return;
}
System.out.println(format(level, format, args));
System.out.flush();
}
public String getDebugf(String format, Object... args) {
return format(DEBUG, format, args);
}
public void debugf(String format, Object... args) {
printf(DEBUG, format, args);
}
public String getInfof(String format, Object... args) {
return format(INFO, format, args);
}
public void infof(String format, Object... args) {
printf(INFO, format, args);
}
public String getWarnf(String format, Object... args) {
return format(WARN, format, args);
}
public void warnf(String format, Object... args) {
printf(WARN, format, args);
}
public String getErrorf(String format, Object... args) {
return format(ERROR, format, args);
}
public void errorf(String format, Object... args) {
printf(ERROR, format, args);
}
public Log setLevel(int level) {
this.level = level;
return this;
}
public Log setLevel(String level) {
String v = level.toLowerCase();
if ("debug".equals(v)) {
this.level = DEBUG;
} else if ("info".equals(v)) {
this.level = INFO;
} else if ("warn".equals(v)) {
this.level = WARN;
} else if ("error".equals(v)) {
this.level = ERROR;
}
return this;
}
}

130
binding/java/src/main/java/org/lionsoul/ip2region/xdb/LongByteArray.java
View File

@ -1,130 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package org.lionsoul.ip2region.xdb;
// xdb byte buffer which used to instead of the byte array
// when the size of the xdb file is greater than 2^32 << 2;
// xdb file v4 is designed to be a maximum of 2^32 bytes in size.
// @Author Leon <chenxin619315@gmail.com>
// @Date 2025/08/22
import java.util.ArrayList;
import java.util.List;
public class LongByteArray {
// byte buffer list
private final List<byte[]> buffs = new ArrayList<byte[]>();
private long length;
public LongByteArray() {
this.length = 0;
}
public LongByteArray(byte[] buff) {
buffs.add(buff);
length = buff.length;
}
// append new buffer
public void append(final byte[] buffer) {
buffs.add(buffer);
length += buffer.length;
}
public long length() {
return length;
}
// internal method to determine the position of the specified offset
private Position determinate(final long offset) {
int index = 0, position = 0, buffLen = buffs.size();
long curIndex = 0;
for (index = 0; index < buffLen; index++) {
final byte[] buff = buffs.get(index);
if (curIndex + buff.length < offset) {
curIndex += buff.length;
continue;
}
// matched and calc the position
position = (int) (offset - curIndex);
break;
}
return new Position(index, position);
}
// Copy from the current buffers to a specified buffer
// from the specified offset with a specified length
public byte[] copy(final long srcPos, final byte[] dest, final int destPos, final int length) {
if (srcPos >= this.length) {
throw new IndexOutOfBoundsException("srcPos exceed the maximum array length `" + this.length + "`");
}
if (destPos + length > dest.length) {
throw new IndexOutOfBoundsException("destPost+length exceed the maximum dest buffer length `" + dest.length + "`");
}
final Position pos = determinate(srcPos);
// copy from the current buffer
final byte[] hBuff = buffs.get(pos.index++);
final int copyLen = Math.min(hBuff.length - pos.offset, length);
System.arraycopy(hBuff, pos.offset, dest, destPos, copyLen);
// check and copy from the rest buffer?
int sPos = destPos + copyLen;
int left = length - copyLen;
while (left > 0) {
final byte[] tBuff = buffs.get(pos.index++);
final int buffLen = tBuff.length;
if (left <= buffLen) {
System.arraycopy(tBuff, 0, dest, sPos, left);
break;
}
System.arraycopy(tBuff, 0, dest, sPos, buffLen);
sPos += buffLen;
left -= buffLen;
}
return dest;
}
// get a byte-buffer from the specified index with a specified length.
// this method will allocate a new byte buffer with length = $length.
public byte[] slice(long offset, int length) {
if (offset + length > this.length) {
throw new IndexOutOfBoundsException("offset+length exceed the maximum array length `" + this.length + "`");
}
final byte[] buffer = new byte[length];
return copy(offset, buffer, 0, length);
}
// get a 4-bytes uint32 integer from the specified index
public long getUint32(long offset) {
final byte[] b = new byte[4];
copy(offset, b, 0, 4);
return LittleEndian.getUint32(b, 0);
}
public int getInt2(long offset) {
final byte[] b = new byte[4];
copy(offset, b, 0, 4);
return LittleEndian.getInt2(b, 0);
}
// position entry class
public static class Position {
public int index;
public int offset;
public Position(int index, int offset) {
this.index = index;
this.offset = offset;
}
}
}

272
binding/java/src/main/java/org/lionsoul/ip2region/xdb/Searcher.java
View File

@ -1,272 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package org.lionsoul.ip2region.xdb;
// xdb searcher (Not thread safe implementation)
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/23
import java.io.IOException;
import java.io.RandomAccessFile;
public class Searcher {
// xdb structure version no
public static final int STRUCTURE_20 = 2;
public static final int STRUCTURE_30 = 3;
// constant defined copied from the xdb maker
public static final int HeaderInfoLength = 256;
public static final int VectorIndexRows = 256;
public static final int VectorIndexCols = 256;
public static final int VectorIndexSize = 8;
// Linux max write / read bytes
public static final int MAX_WRITE_BYTES = 0x7ffff000;
// ip version
private final Version version;
// random access file handle for file-based search
private final RandomAccessFile handle;
private int ioCount = 0;
// vector index.
// use the byte[] instead of VectorIndex entry array to keep
// the minimal memory allocation.
private final byte[] vectorIndex;
// xdb content buffer, used for in-memory search.
// @Note: use the LongByteArray instead since 2025/08/22
// private final byte[] contentBuff;
private final LongByteArray contentBuff;
// --- static method to create searchers
public static Searcher newWithFileOnly(Version version, String dbPath) throws IOException {
return new Searcher(version, dbPath, null, null);
}
public static Searcher newWithVectorIndex(Version version, String dbPath, byte[] vectorIndex) throws IOException {
return new Searcher(version, dbPath, vectorIndex, null);
}
public static Searcher newWithBuffer(Version version, LongByteArray cBuff) throws IOException {
return new Searcher(version, null, null, cBuff);
}
// --- End of creator
public Searcher(Version version, String dbFile, byte[] vectorIndex, LongByteArray cBuff) throws IOException {
this.version = version;
if (cBuff != null) {
this.handle = null;
this.vectorIndex = null;
this.contentBuff = cBuff;
} else {
this.handle = new RandomAccessFile(dbFile, "r");
this.vectorIndex = vectorIndex;
this.contentBuff = null;
}
}
public void close() throws IOException {
if (this.handle != null) {
this.handle.close();
}
}
public Version getIPVersion() {
return version;
}
public int getIOCount() {
return ioCount;
}
public String search(String ipStr) throws Exception {
return search(Util.parseIP(ipStr));
}
public String search(byte[] ip) throws IOException, InetAddressException {
// ip version check
if (ip.length != version.bytes) {
throw new InetAddressException("invalid ip address ("+version.name+" expected)");
}
// reset the global counter
this.ioCount = 0;
// locate the segment index block based on the vector index
long sPtr = 0, ePtr = 0;
int il0 = (int) (ip[0] & 0xFF);
int il1 = (int) (ip[1] & 0xFF);
int idx = il0 * VectorIndexCols * VectorIndexSize + il1 * VectorIndexSize;
// System.out.printf("il0: %d, il1: %d, idx: %d\n", il0, il1, idx);
if (vectorIndex != null) {
sPtr = LittleEndian.getUint32(vectorIndex, idx);
ePtr = LittleEndian.getUint32(vectorIndex, idx + 4);
} else if (contentBuff != null) {
sPtr = contentBuff.getUint32(HeaderInfoLength + idx);
ePtr = contentBuff.getUint32(HeaderInfoLength + idx + 4);
} else {
final byte[] buff = new byte[VectorIndexSize];
read(HeaderInfoLength + idx, buff);
sPtr = LittleEndian.getUint32(buff, 0);
ePtr = LittleEndian.getUint32(buff, 4);
}
// System.out.printf("sPtr: %d, ePtr: %d\n", sPtr, ePtr);
// binary search the segment index block to get the region info
final int bytes = ip.length, dBytes = ip.length << 1;
final int segIndexSize = version.segmentIndexSize;
final byte[] buff = new byte[segIndexSize];
int dataLen = -1;
long dataPtr = -1, l = 0, h = (ePtr - sPtr) / segIndexSize;
while (l <= h) {
long m = (l + h) >> 1;
long p = sPtr + m * segIndexSize;
// read the segment index
read(p, buff);
if (version.ipSubCompare(ip, buff, 0) < 0) {
h = m - 1;
} else if (version.ipSubCompare(ip, buff, bytes) > 0) {
l = m + 1;
} else {
dataLen = LittleEndian.getInt2(buff, dBytes);
dataPtr = LittleEndian.getUint32(buff, dBytes + 2);
break;
}
}
// empty match interception
// System.out.printf("dataLen: %d, dataPtr: %d\n", dataLen, dataPtr);
if (dataPtr < 0) {
return null;
}
// load and return the region data
final byte[] regionBuff = new byte[dataLen];
read(dataPtr, regionBuff);
return new String(regionBuff, "utf-8");
}
protected void read(long offset, byte[] buffer) throws IOException {
// check the in-memory buffer first
if (contentBuff != null) {
contentBuff.copy(offset, buffer, 0, buffer.length);
return;
}
// read from the file handle
assert handle != null;
handle.seek(offset);
this.ioCount++;
int rLen = handle.read(buffer);
if (rLen != buffer.length) {
throw new IOException("incomplete read: read bytes should be " + buffer.length);
}
}
// --- static util function
public static Header loadHeader(RandomAccessFile handle) throws IOException {
handle.seek(0);
final byte[] buff = new byte[HeaderInfoLength];
handle.read(buff);
return new Header(buff);
}
public static Header loadHeaderFromFile(String dbPath) throws IOException {
final RandomAccessFile handle = new RandomAccessFile(dbPath, "r");
final Header header = loadHeader(handle);
handle.close();
return header;
}
public static byte[] loadVectorIndex(RandomAccessFile handle) throws IOException {
handle.seek(HeaderInfoLength);
int len = VectorIndexRows * VectorIndexCols * VectorIndexSize;
final byte[] buff = new byte[len];
int rLen = handle.read(buff);
if (rLen != len) {
throw new IOException("incomplete read: read bytes should be " + len);
}
return buff;
}
public static byte[] loadVectorIndexFromFile(String dbPath) throws IOException {
final RandomAccessFile handle = new RandomAccessFile(dbPath, "r");
final byte[] vIndex = loadVectorIndex(handle);
handle.close();
return vIndex;
}
public static LongByteArray loadContent(RandomAccessFile handle) throws IOException {
handle.seek(0);
// check the length and do the buff load
long toRead = handle.length();
final LongByteArray byteArray = new LongByteArray();
while (toRead > 0) {
final byte[] buff = new byte[(int) Math.min(toRead, MAX_WRITE_BYTES)];
final int rLen = handle.read(buff);
if (rLen != buff.length) {
throw new IOException("incomplete read: read bytes should be " + buff.length + ", got `" + rLen + "`");
}
byteArray.append(buff);
toRead -= rLen;
}
return byteArray;
}
public static LongByteArray loadContentFromFile(String dbPath) throws IOException {
final RandomAccessFile handle = new RandomAccessFile(dbPath, "r");
final LongByteArray content = loadContent(handle);
handle.close();
return content;
}
// --- verify util function
// Verify if the current Searcher could be used to search the specified xdb file.
// Why do we need this check ?
// The future features of the xdb impl may cause the current searcher not able to work properly.
//
// @Note: You Just need to check this ONCE when the service starts
// Or use another process (eg, A command) to check once Just to confirm the suitability.
public static void verify(RandomAccessFile handle) throws IOException, XdbException {
final Header header = loadHeader(handle);
// get the runtime ptr bytes
int runtimePtrBytes = 0;
if (header.version == STRUCTURE_20) {
runtimePtrBytes = 4;
} else if (header.version == STRUCTURE_30) {
runtimePtrBytes = header.runtimePtrBytes;
} else {
throw new XdbException("invalid structure version `" + header.version + "`");
}
// 1, confirm the xdb file size
// to ensure that the maximum file pointer does not overflow
final long maxFilePtr = (1L << (runtimePtrBytes * 8)) - 1;
if (handle.length() > maxFilePtr) {
throw new XdbException("xdb file exceeds the maximum supported bytes: "+maxFilePtr+"");
}
}
public static void verifyFromFile(String dbFile) throws IOException, XdbException {
final RandomAccessFile handle = new RandomAccessFile(dbFile, "r");
verify(handle);
handle.close();
}
}

82
binding/java/src/main/java/org/lionsoul/ip2region/xdb/Util.java
View File

@ -1,82 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
//
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/07/14
package org.lionsoul.ip2region.xdb;
import java.net.InetAddress;
import java.net.UnknownHostException;
public class Util
{
// parse the specified IP address and return its bytes.
// returns: byte[4] for IPv4 and byte[16] for IPv6 and the bytes should be in Big endian order.
public static byte[] parseIP(String ip) throws InetAddressException {
try {
return InetAddress.getByName(ip).getAddress();
} catch (UnknownHostException e) {
throw new InetAddressException("invalid ip address `"+ip+"`");
}
}
// convert the byte[] ip to string ip address
public static String ipToString(final byte[] ip) {
if (ip.length != 4 && ip.length != 16) {
return String.format("invalid-ip-address-length: %d", ip.length);
}
try {
return InetAddress.getByAddress(ip).getHostAddress();
} catch (UnknownHostException e) {
return String.format("invalid-ip-address `%s`", ipJoin(ip));
}
}
// implode the byte[] ip with its byte value.
public static String ipJoin(byte[] ip) {
return bytesToString(ip, 0, ip.length);
}
public static String bytesToString(byte[] buff, int offset, int length) {
final StringBuffer sb = new StringBuffer();
sb.append("[");
for (int i = 0; i < length; i++) {
if (i > 0) {
sb.append(',');
}
sb.append((buff[offset+i] & 0xFF));
}
sb.append("]");
return sb.toString();
}
// compare two byte ip
// Returns: -1 if ip1 < ip2, 0 if ip1 == ip2, 1 if ip1 > ip2
public static int ipCompare(byte[] ip1, byte[] ip2) {
return ipSubCompare(ip1, ip2, 0);
}
// compare the ip with the ip in the buffer start from offset
// Returns: -1 if ip < buff[offset], 0 if ip == buff[offset], 1 if ip > buff[offset]
public static int ipSubCompare(byte[] ip, byte[] buff, int offset) {
for (int i = 0; i < ip.length; i++) {
// covert the byte to int to sure the uint8 attribute
final int i1 = (int)(ip[i] & 0xFF);
final int i2 = (int)(buff[offset+i] & 0xFF);
if (i1 < i2) {
return -1;
}
if (i1 > i2) {
return 1;
}
}
return 0;
}
}

83
binding/java/src/main/java/org/lionsoul/ip2region/xdb/Version.java
View File

@ -1,83 +0,0 @@
// Copyright 2025 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package org.lionsoul.ip2region.xdb;
// IP version abstract manager (IPv4 & IPv6)
// @Author Lion <chenxin619315@gmail.com>
// @Date 2025/09/10
public abstract class Version {
public static final int IPv4VersionNo = 4;
public static final int IPv6VersionNo = 6;
public static final IPv4 IPv4 = new IPv4();
public static final IPv6 IPv6 = new IPv6();
// version id and name
public final int id;
public final String name;
// the numbers of bytes for one IP
public final int bytes;
// segment index size (bytes)
public final int segmentIndexSize;
public Version(int id, String name, int bytes, int segmentIndexSize) {
this.id = id;
this.name = name;
this.bytes = bytes;
this.segmentIndexSize = segmentIndexSize;
}
// encode the specified IP bytes to the specified buffer
public abstract int putBytes(byte[] buff, int offset, byte[] ip);
// compare the two IPs with the current version.
// Returns: -1 if ip1 < ip2, 0 if ip1 == ip2, 1 if ip1 > ip2
public int ipCompare(byte[] ip1, byte[] ip2) {
return ipSubCompare(ip1, ip2, 0);
}
// @see ipCompare
public abstract int ipSubCompare(byte[] ip1, byte[] buff, int offset);
// parse the version from an name
public static final Version fromName(String name) throws Exception {
final String n = name.toUpperCase();
if (n.equals("V4") || n.equals("IPV4")) {
return IPv4;
} else if (n.equals("V6") || n.equals("IPV6")) {
return IPv6;
} else {
throw new Exception("invalid version name `"+name+"`");
}
}
// parse the version from header
public static final Version fromHeader(Header header) throws XdbException {
// Old 2.0 structure with IPv4 supports ONLY.
if (header.version == Searcher.STRUCTURE_20) {
return IPv4;
}
// structure 3.0 after IPv6 supporting
if (header.version != Searcher.STRUCTURE_30) {
throw new XdbException("invalid xdb structure version `"+header.version+"`");
}
if (header.ipVersion == IPv4VersionNo) {
return IPv4;
} else if (header.ipVersion == IPv6VersionNo) {
return IPv6;
} else {
throw new XdbException("invalid ip version number `" + header.ipVersion + "`");
}
}
@Override public String toString() {
return String.format("{Id:%d, Name:%s, Bytes:%d, IndexSize: %d}", id, name, bytes, segmentIndexSize);
}
}

13
binding/java/src/main/java/org/lionsoul/ip2region/xdb/XdbException.java
View File

@ -1,13 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
package org.lionsoul.ip2region.xdb;
public class XdbException extends Exception {
public XdbException(String str) {
super(str);
}
}

44
binding/java/src/test/java/org/lionsoul/ip2region/xdb/IPv4Test.java
View File

@ -1,44 +0,0 @@
package org.lionsoul.ip2region.xdb;
import org.junit.Test;
public class IPv4Test {
private static final Log log = Log.getLogger(IPv4Test.class);
@Test
public void testIpSubCompare() throws InetAddressException {
final byte[] sip = Util.parseIP("0.255.255.255");
final byte[] eip = Util.parseIP("1.0.0.2");
final byte[] buff = new byte[Version.IPv4.segmentIndexSize];
Version.IPv4.putBytes(buff, 0, sip);
Version.IPv4.putBytes(buff, 4, eip);
log.infof("bytesToString(buff): %s", Util.bytesToString(buff, 0, 8));
final byte[] ip = Util.parseIP("1.0.0.0");
// compare the sip
log.infof("ipSubCompare(%s, %s): %d",
Util.ipToString(ip),
Util.bytesToString(buff, 0, 4),
Util.ipSubCompare(ip, buff, 0)
);
log.infof("IPv4.ipSubCompare(%s, %s): %d",
Util.ipToString(ip),
Util.bytesToString(buff, 0, 4),
Version.IPv4.ipSubCompare(ip, buff, 0)
);
// compare the eip
log.infof("ipSubCompare(%s, %s): %d",
Util.ipToString(ip),
Util.bytesToString(buff, 4, 4),
Util.ipSubCompare(ip, buff, 4)
);
log.infof("IPv4.ipSubCompare(%s, %s): %d",
Util.ipToString(ip),
Util.bytesToString(buff, 4, 4),
Version.IPv4.ipSubCompare(ip, buff, 4)
);
}
}

36
binding/java/src/test/java/org/lionsoul/ip2region/xdb/LittleEndianTest.java
View File

@ -1,36 +0,0 @@
package org.lionsoul.ip2region.xdb;
import static org.junit.Assert.assertEquals;
import org.junit.Test;
public class LittleEndianTest {
private static final Log log = Log.getLogger(LittleEndianTest.class).setLevel(Log.DEBUG);
@Test
public void testAll() {
final byte[] buff = new byte[14];
// encode
// do the put
LittleEndian.put(buff, 0, 1L, 4);
LittleEndian.put(buff, 4, 2L, 4);
// putUint32
LittleEndian.putInt2(buff, 8, 24);
LittleEndian.putUint32(buff, 10, 1024L);
// decode
assertEquals(LittleEndian.getUint32(buff, 0), 1);
assertEquals(LittleEndian.getUint32(buff, 4), 2);
assertEquals(LittleEndian.getInt2(buff, 8), 24);
assertEquals(LittleEndian.getUint32(buff, 10), 1024);
log.debugf("uint32(buff, 0): %d", LittleEndian.getUint32(buff, 0));
log.debugf("uint32(buff, 4): %d", LittleEndian.getUint32(buff, 4));
log.debugf("int2(buff, 8): %d", LittleEndian.getInt2(buff, 8));
log.debugf("uint32(buff, 10): %d", LittleEndian.getUint32(buff, 10));
}
}

61
binding/java/src/test/java/org/lionsoul/ip2region/xdb/UtilTest.java
View File

@ -1,61 +0,0 @@
package org.lionsoul.ip2region.xdb;
import org.junit.Test;
public class UtilTest {
private static final Log log = Log.getLogger(UtilTest.class).setLevel(Log.DEBUG);
@Test
public void testCheckIP() throws InetAddressException {
final String[] ips = new String[]{
"192.168.1.102",
"219.133.111.87",
"::",
"3000::",
"::1001:ffff",
"2001:2:0:ffff:ffff:ffff:ffff:ffff",
"::ffff:114.114.114.114"
};
for (String ip : ips) {
final byte[] ipBytes = Util.parseIP(ip);
log.debugf("%s(v=%s) => %s", ip, Util.ipJoin(ipBytes), Util.ipToString(ipBytes));
}
}
@Test
public void testIpCompare() throws InetAddressException {
final String[][] ipPairs = new String[][]{
{"1.0.0.0", "1.0.0.1"},
{"192.168.1.101", "192.168.1.90"},
{"219.133.111.87", "114.114.114.114"},
{"2000::", "2000:ffff:ffff:ffff:ffff:ffff:ffff:ffff"},
{"2001:4:112::", "2001:4:112:ffff:ffff:ffff:ffff:ffff"},
{"ffff::", "2001:4:ffff:ffff:ffff:ffff:ffff:ffff"}
};
for (String[] ips : ipPairs) {
final byte[] ip1 = Util.parseIP(ips[0]);
final byte[] ip2 = Util.parseIP(ips[1]);
log.debugf("compare(%s, %s): %d", ips[0], ips[1], Util.ipCompare(ip1, ip2));
}
}
@Test
public void testIpSubCompare() throws InetAddressException {
final String[][] ipPairs = new String[][] {
{"1.0.0.0", "1.0.0.1"},
{"192.168.1.100", "192.168.2.100"},
{"10.100.1.10", "11.100.2.10"},
{"11.100.1.10", "10.100.2.10"}
};
for (final String[] ips : ipPairs) {
final byte[] ip1 = Util.parseIP(ips[0]);
final byte[] ip2 = Util.parseIP(ips[1]);
log.debugf("ipSubCompare(%s, %s): %d", Util.ipToString(ip1), Util.ipToString(ip2), Util.ipSubCompare(ip1, ip2, 0));
}
}
}

41
binding/java/src/test/java/org/lionsoul/ip2region/xdb/VersionTest.java
View File

@ -1,41 +0,0 @@
package org.lionsoul.ip2region.xdb;
import static org.junit.Assert.assertEquals;
import org.junit.Test;
public class VersionTest {
private static final Log log = Log.getLogger(VersionTest.class).setLevel(Log.DEBUG);
@Test
public void testFromName() throws Exception {
final String[] vers = new String[]{"IPv4", "IPv6"};
final Version v4 = Version.fromName(vers[0]);
assertEquals(v4.name, vers[0]);
final Version v6 = Version.fromName(vers[1]);
assertEquals(v6.name, vers[1]);
log.debugf("v4: %s", v4);
log.debugf("v6: %s", v6);
}
@Test
public void testFromHeader() throws XdbException {
// create the header buffer
final byte[] buff1 = new byte[Searcher.HeaderInfoLength];
// structure version
LittleEndian.put(buff1, 0, 2, 2);
LittleEndian.put(buff1, 16, Version.IPv4VersionNo, 2);
final byte[] buff2 = new byte[Searcher.HeaderInfoLength];
LittleEndian.put(buff2, 0, 3, 2);
LittleEndian.put(buff2, 16, Version.IPv6VersionNo, 2);
final Version ver1 = Version.fromHeader(new Header(buff1));
final Version ver2 = Version.fromHeader(new Header(buff2));
log.debugf("ver1: %s", ver1.toString());
log.debugf("ver2: %s", ver2.toString());
}
}

153
binding/lua/ReadMe.md
View File

@ -1,153 +0,0 @@
# ip2region xdb lua 查询客户端实现
#### 备注：请优先使用 lua_c 扩展 xdb 查询客户端，性能比纯 lua 实现的要快很多！！！
# 使用方式
### 完全基于文件的查询
```lua
local xdb = require("xdb_searcher")
-- 1、从 db_path 创建基于文件的 xdb 查询对象
local db_path = "ip2region.xdb file path"
local searcher, err = xdb.new_with_file_only(db_path)
if err ~= nil then
print(string.format("failed to create searcher: %s", err))
return
end
-- 2、调用查询 API 进行查询
local ip_str = "1.2.3.4"
local s_time = xdb.now()
region, err = searcher:search(ip_str)
if err ~= nil then
print(string.format("failed to search(%s): %s", ip_str, err))
return
end
-- 备注：并发使用，每个协程需要创建单独的 xdb 查询对象
print(string.format("{region: %s, took: %.5f μs}", region, xdb.now() - s_time))
```
### 缓存 `VectorIndex` 索引
如果你的 `lua` 母环境支持，可以预先加载 vectorIndex 缓存，然后做成全局变量，每次创建 Searcher 的时候使用全局的 vectorIndex，可以减少一次固定的 IO 操作从而加速查询，减少 io 压力。
```lua
local xdb = require("xdb_searcher")
local db_path = "ip2region.xdb file path"
-- 1、从指定的 db_path 加载 VectorIndex 缓存，把下述的 v_index 对象做成全局变量。
-- vectorIndex 加载一次即可，建议在服务启动的时候加载为全局对象。
v_index, err = xdb.load_vector_index(db_path)
if err ~= nil then
print(string.format("failed to load vector index from '%s'", db_path))
return
end
-- 2、使用全局的 v_index 创建带 VectorIndex 缓存的查询对象。
searcher, err = xdb.new_with_vector_index(db_path, v_index)
if err ~= nil then
print(string.format("failed to create vector index searcher: %s", err))
return
end
-- 3、调用查询 API
local ip_str = "1.2.3.4"
local s_time = xdb.now()
region, err = searcher:search(ip_str)
if err ~= nil then
print(string.format("failed to search(%s): %s", ip_str, err))
return
end
-- 备注：并发使用，每个协程需要创建单独的 xdb 查询对象，但是共享全局的 v_index 对象
print(string.format("{region: %s, took: %.5f μs}", region, xdb.now() - s_time))
```
### 缓存整个 `xdb` 数据
如果你的 `lua` 母环境支持，可以预先加载整个 xdb 的数据到内存，这样可以实现完全基于内存的查询，类似之前的 memory search 查询。
```lua
local xdb = require("xdb_searcher")
local db_path = "ip2region.xdb file path"
-- 1、从指定的 db_path 加载整个 xdb 到内存。
-- xdb内容加载一次即可，建议在服务启动的时候加载为全局对象。
content = xdb.load_content(db_path)
if content == nil then
print(string.format("failed to load xdb content from '%s'", db_path))
return
end
-- 2、使用全局的 content 创建带完全基于内存的查询对象。
searcher, err = xdb.new_with_buffer(content)
if err ~= nil then
print(string.format("failed to create content buffer searcher: %s", err))
return
end
-- 3、调用查询 API
local ip_str = "1.2.3.4"
local s_time = xdb.now()
region, err = searcher:search(ip_str)
if err ~= nil then
print(string.format("failed to search(%s): %s", ip_str, err))
return
end
-- 备注：并发使用，用 xdb 整个缓存创建的查询对象可以安全的用于并发。
-- 建议在服务启动的时候创建好全局的 searcher 对象，然后全局并发使用。
print(string.format("{region: %s, took: %.5f μs}", region, xdb.now() - s_time))
```
# 查询测试
通过 `search_test.lua` 脚本来进行查询测试：
```bash
➜ lua git:(lua_binding) ✗ lua search_test.lua
lua search_test.lua [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：使用默认的 data/ip2region.xdb 进行查询测试：
```bash
➜ lua git:(lua_binding) ✗ lua search_test.lua --db=../../data/ip2region.xdb --cache-policy=vectorIndex
ip2region xdb searcher test program, cachePolicy: vectorIndex
type 'quit' to exit
ip2region>> 1.2.3.4
{region: 美国|0|华盛顿|0|谷歌, io_count: 7, took: 0μs}
ip2region>>
```
输入 ip 即可进行查询测试。也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的效率。
# bench 测试
通过 `bench_test.lua` 脚本来进行自动 bench 测试，一方面确保 `xdb` 文件没有错误，另一方面通过大量的查询测试平均查询性能：
```bash
➜ lua git:(lua_binding) ✗ lua bench_test.lua
lua bench_test.lua [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：通过默认的 data/ip2region.xdb 和 data/ip.merge.txt 来进行 bench 测试：
```bash
➜ lua git:(lua_binding) ✗ lua bench_test.lua --db=../../data/ip2region.xdb --src=../../data/ip.merge.txt --cache-policy=vectorIndex
Bench finished, {cachePolicy: vectorIndex, total: 3417955, took: 29.000 s, cost: 7.899 μs/op}
```
可以通过设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 三种不同的缓存实现的的性能。
@Note：请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件的相同的源文件。

175
binding/lua/bench_test.lua
View File

@ -1,175 +0,0 @@
-- Copyright 2022 The Ip2Region Authors. All rights reserved.
-- Use of this source code is governed by a Apache2.0-style
-- license that can be found in the LICENSE file.
--
-- ---
-- @Author Lion <chenxin619315@gmail.com>
-- @Date 2022/06/30
-- set the package to load the current xdb_searcher.so
package.path = "./?.lua" .. package.path
package.cpath = "./?.so" .. package.cpath
local xdb = require("xdb_searcher")
function printHelp()
print("lua bench_test.lua [command options]")
print("options: ")
print(" --db string ip2region binary xdb file path")
print(" --src string source ip text file path")
print(" --cache-policy string cache policy: file/vectorIndex/content")
end
if #arg < 2 then
printHelp(arg)
return
end
-- parser the command line args
local dbFile, srcFile = "", ""
local cachePolicy = "vectorIndex"
for _, r in ipairs(arg) do
if string.len(r) < 5 then
goto continue
end
if string.sub(r, 1, 2) ~= "--" then
goto continue
end
for k, v in string.gmatch(string.sub(r, 3), "([^=]+)=([^%s]+)") do
if k == "db" then
dbFile = v
elseif k == "src" then
srcFile = v
elseif k == "cache-policy" then
cachePolicy = v
else
print(string.format("undefined option `%s`", r))
return
end
-- break the match iterate
break
end
-- continue this loop
::continue::
end
-- print(string.format("dbFile=%s, srcFile=%s, cachePolicy=%s", dbFile, srcFile, cachePolicy))
if string.len(dbFile) < 2 or string.len(srcFile) < 2 then
printHelp()
return
end
-- create the searcher based on the cache-policy
local searcher, v_index, content
if cachePolicy == "file" then
searcher, err = xdb.new_with_file_only(dbFile)
if err ~= nil then
print(string.format("failed to create searcher: %s", err))
return
end
elseif cachePolicy == "vectorIndex" then
v_index, err = xdb.load_vector_index(dbFile)
if err ~= nil then
print(string.format("failed to load vector index: %s", err))
return
end
searcher, err = xdb.new_with_vector_index(dbFile, v_index)
if err ~= nil then
print(string.format("failed to create vector index searcher: %s", err))
return
end
elseif cachePolicy == "content" then
content, err = xdb.load_content(dbFile)
if err ~= nil then
print(string.format("failed to load xdb content: %s", err))
return
end
searcher, err = xdb.new_with_buffer(content)
if err ~= nil then
print(string.format("failed to create content buffer searcher: %s", err))
return
end
else
print(string.format("undefined cache-policy `%s`", cachePolicy))
return
end
-- do the bench test
local handle = io.open(srcFile, "r")
if handle == nil then
print(string.format("failed to open src text file `%s`", handle))
return
end
local lines = handle:lines()
local sip_str, eip_str, s_region, region = "", "", "", ""
local sip, mip, eip, err = 0, 0, 0, 0
local count, t_time, c_time = 0, 0, 0
local s_time = xdb.now()
for l in lines do
if string.len(l) < 1 then
goto continue
end
for v1, v2, v3 in string.gmatch(l, "([%d%.]+)|([%d%.]+)|([^\n]+)") do
-- print(sip_str, eip_str, region)
sip_str = v1
eip_str = v2
s_region = v3
break
end
sip, err = xdb.check_ip(sip_str)
if err ~= nil then
print(string.format("invalid start ip `%s`", sip_str))
return
end
eip, err = xdb.check_ip(eip_str)
if err ~= nil then
print(string.format("invalid end ip `%s`", sip_str))
return
end
if sip > eip then
print(string.format("start ip(%s) should not be greater than end ip(%s)\n", sip_str, eip_str))
return
end
mip = (sip + eip) >> 1
for _, ip in ipairs({sip, (sip + mip) >> 1, mip, (mip + eip) >> 1, eip}) do
t_time = xdb.now()
region, err = searcher:search(ip)
c_time = c_time + xdb.now() - t_time
if err ~= nil then
print(string.format("failed to search ip `%s`", xdb.long2ip(ip)))
return
end
-- check the region
if region ~= s_region then
print(string.format("failed search(%s) with (%s != %s)\n", xdb.long2ip(ip), region, s_region))
return
end
count = count + 1
end
::continue::
end
-- resource cleanup
searcher:close()
-- print the stats
local avg_costs = 0
if count > 0 then
avg_costs = c_time / count
end
print(string.format("Bench finished, {cachePolicy: %s, total: %d, took: %.3f s, cost: %.3f μs/op}",
cachePolicy, count, (xdb.now() - s_time)/1e6, c_time / count))

137
binding/lua/search_test.lua
View File

@ -1,137 +0,0 @@
-- Copyright 2022 The Ip2Region Authors. All rights reserved.
-- Use of this source code is governed by a Apache2.0-style
-- license that can be found in the LICENSE file.
--
-- ---
-- @Author Lion <chenxin619315@gmail.com>
-- @Date 2022/06/30
-- set the package to load the current xdb_searcher.so
package.path = "./?.lua" .. package.path
package.cpath = "./?.so" .. package.cpath
local xdb = require("xdb_searcher")
function printHelp()
print("lua search_test.lua [command options]")
print("options: ")
print(" --db string ip2region binary xdb file path")
print(" --cache-policy string cache policy: file/vectorIndex/content")
end
if #arg < 2 then
printHelp(arg)
return
end
-- parser the command line args
local dbFile = ""
local cachePolicy = "vectorIndex"
for _, r in ipairs(arg) do
if string.len(r) < 5 then
goto continue
end
if string.sub(r, 1, 2) ~= "--" then
goto continue
end
for k, v in string.gmatch(string.sub(r, 3), "([^=]+)=([^%s]+)") do
if k == "db" then
dbFile = v
elseif k == "cache-policy" then
cachePolicy = v
else
print(string.format("undefined option `%s`", r))
return
end
-- break the match iterate
break
end
-- continue this loop
::continue::
end
-- print(string.format("dbFile=%s, cachePolicy=%s", dbFile, cachePolicy))
if string.len(dbFile) < 2 then
printHelp()
return
end
-- create the searcher based on the cache-policy
local searcher, v_index, content
if cachePolicy == "file" then
searcher, err = xdb.new_with_file_only(dbFile)
if err ~= nil then
print(string.format("failed to create searcher: %s", err))
return
end
elseif cachePolicy == "vectorIndex" then
v_index, err = xdb.load_vector_index(dbFile)
if err ~= nil then
print(string.format("failed to load vector index: %s", err))
return
end
searcher, err = xdb.new_with_vector_index(dbFile, v_index)
if err ~= nil then
print(string.format("failed to create vector index searcher: %s", err))
return
end
elseif cachePolicy == "content" then
content, err = xdb.load_content(dbFile)
if err ~= nil then
print(string.format("failed to load xdb content: %s", err))
return
end
searcher, err = xdb.new_with_buffer(content)
if err ~= nil then
print(string.format("failed to create content buffer searcher: %s", err))
return
end
else
print(string.format("undefined cache-policy `%s`", cachePolicy))
return
end
-- do the search
print(string.format([[
ip2region xdb searcher test program, cachePolicy: %s
type 'quit' to exit]], cachePolicy))
local region, err = "", nil
local ip_int, s_time, c_time = 0, 0, 0
while ( true ) do
io.write("ip2region>> ");
io.input(io.stdin);
local line = io.read();
if (line == nil) then
break
end
if ( line == "quit" ) then
break
end
ip_int, err = xdb.check_ip(line)
if err ~= nil then
print(string.format("invalid ip address `%s`", line))
goto continue
end
-- do the search
s_time = xdb.now()
region, err = searcher:search(line)
if err ~= nil then
print(string.format("{err: %s, io_count: %d}", err, searcher:get_io_count()))
else
c_time = xdb.now() - s_time
print(string.format("{region: %s, io_count: %d, took: %dμs}", region, searcher:get_io_count(), c_time))
end
::continue::
end
-- resource cleanup
searcher:close()

98
binding/lua/util_test.lua
View File

@ -1,98 +0,0 @@
-- Copyright 2022 The Ip2Region Authors. All rights reserved.
-- Use of this source code is governed by a Apache2.0-style
-- license that can be found in the LICENSE file.
--
-- ---
-- @Author Lion <chenxin619315@gmail.com>
-- @Date 2022/07/05
-- set the package path
package.path = "./?.lua"
package.cpath = "./?.so"
local xdb = require("xdb_searcher")
---- ip checking testing
print("--- testing check_ip and long2ip ... ")
local ip_list = {
"1.2.3.4", "192.168.2.3", "120.24.78.129", "255.255.255.0",
"256.7.12.9", "12.56.78.320", "32.12.45.192", "222.221.220.219",
"192.168.1.101 ", "132.96.12.98a", "x23.12.2.12"
}
local s_time = xdb.now()
for _, ip_src in ipairs(ip_list) do
ip, err = xdb.check_ip(ip_src)
if err ~= nil then
print(string.format("invalid ip address `%s`: %s", ip_src, err))
else
ip_dst = xdb.long2ip(ip)
io.write(string.format("long(%-15s)=%10d, long2ip(%-10d)=%-15s", ip_src, ip, ip, ip_dst))
if ip_src ~= ip_dst then
print(" --[Failed]")
else
print(" --[Ok]")
end
end
end
---- buffer loading test
print("\n--- testing load header ... ")
header, err = xdb.load_header("../../data/ip2region.xdb")
if err ~= nil then
print("failed to load header: ", err)
else
print("xdb header buffer loaded")
local tpl = [[
header: {
version: %d
index_policy: %d
created_at: %d
start_index_ptr: %d
end_index_ptr: %d
}]]
print(string.format(tpl,
header["version"], header["index_policy"],
header["created_at"], header["start_index_ptr"], header["end_index_ptr"])
)
end
print("\n--- testing load vector index ... ")
v_index, err = xdb.load_vector_index("../../data/ip2region.xdb")
if err ~= nil then
print("failed to load vector index: ", err)
else
print("xdb vector index buffer loaded")
end
print("\n--- testing load content buffer ... ")
c_buffer, err = xdb.load_content("../../data/ip2region.xdb")
if err ~= nil then
print("failed to load content: ", err)
else
print("xdb content buffer loaded")
end
print("\n--- testing search ... ")
local ip_str = "1.2.3.4"
searcher, err = xdb.new_with_file_only("../../data/ip2region.xdb")
local t_start = xdb.now()
region, err = searcher:search(ip_str)
if err ~= nil then
print(string.format("search(%s) failed: %s", ip_str, err))
else
local c_time = xdb.now() - t_start
print(string.format("search(%s): {region=%s, io_count: %d, took: %dμs, err=%s}",
ip_str, region, searcher:get_io_count(), c_time, err))
print(string.format("searcher.tostring=%s", searcher))
end
searcher:close()
print("")
print(string.format("all tests done, elapsed %d μs", xdb.now() - s_time))

334
binding/lua/xdb_searcher.lua
View File

@ -1,334 +0,0 @@
-- Copyright 2022 The Ip2Region Authors. All rights reserved.
-- Use of this source code is governed by a Apache2.0-style
-- license that can be found in the LICENSE file.
--
-- ---
-- @Author Lion <chenxin619315@gmail.com>
-- @Date 2022/07/05
-- constants define
local HeaderInfoLength = 256
local VectorIndexRows = 256
local VectorIndexCols = 256
local VectorIndexSize = 8
local SegmentIndexSize = 14
local VectorIndexLength = 524288
local _M = {
-- xdb file handle
handle = nil,
-- header info
header = nil,
io_count = 0,
-- vector index
vector_index = nil,
-- xdb content buffer
content_buff = nil
}
-- index and to string attribute set
_M.__index = _M
_M.__tostring = function(self)
return "xdb searcher object (lua)"
end
-- construct functions
function newBase(dbPath, vIndex, cBuffer)
local obj = setmetatable({}, _M)
if cBuffer ~= nil then
obj.io_count = 0
obj.vector_index = nil
obj.content_buff = cBuffer
else
obj.io_count = 0
obj.vector_index = vIndex
obj.handle = io.open(dbPath, "r")
if obj.handle == nil then
return nil, string.format("failed to open xdb file `%s`", dbPath)
end
end
return obj, nil
end
function _M.new_with_file_only(dbPath)
return newBase(dbPath, nil, nil)
end
function _M.new_with_vector_index(dbPath, vIndex)
return newBase(dbPath, vIndex, nil)
end
function _M.new_with_buffer(cBuffer)
return newBase(nil, nil, cBuffer)
end
-- End of constructors
-- object api impl, must call via ':'
function _M:search(ip_src)
-- check and convert string ip to long ip
local t, ip = type(ip_src), 0
if t == nil then
return "", string.format("invalid ip address `%s`", ip_src)
elseif t == "string" then
ip, err = self.check_ip(ip_src)
if err ~= nil then
return "", string.format("check ip `%s`: %s", ip_src, err)
end
elseif t ~= "number" then
return "", "invalid number or string ip"
else
-- use the original value
ip = ip_src
end
-- reset the global counter
-- and global resource local cache
self.io_count = 0
local vector_index = self.vector_index
local content_buff = self.content_buff
local read_data = self.read
-- locate the segment index based on the vector index
local il0 = (ip >> 24) & 0xFF
local il1 = (ip >> 16) & 0xFF
local idx = il0 * VectorIndexCols * VectorIndexSize + il1 * VectorIndexSize
local s_ptr, e_ptr = 0, 0
if vector_index ~= nil then
s_ptr = getLong(vector_index, idx + 1)
e_ptr = getLong(vector_index, idx + 5)
elseif content_buff ~= nil then
s_ptr = getLong(content_buff, HeaderInfoLength + idx + 1)
e_ptr = getLong(content_buff, HeaderInfoLength + idx + 5)
else
-- load from the file
buff, err = read_data(self, HeaderInfoLength + idx, SegmentIndexSize)
if err ~= nil then
return "", string.format("read buffer: %s", err)
end
s_ptr = getLong(buff, 1)
e_ptr = getLong(buff, 5)
end
-- print(string.format("s_ptr: %d, e_ptr: %d", s_ptr, e_ptr))
-- binary search to get the data
local data_ptr, data_len, p = 0, 0, 0
local sip, eip, err, buff = 0, 0, ""
local l, m, h = 0, 0, (e_ptr - s_ptr) / SegmentIndexSize
while l <= h do
m = (l + h) >> 1
p = s_ptr + m * SegmentIndexSize
-- read the segment index
buff, err = read_data(self, p, SegmentIndexSize)
if err ~= nil then
return "", string.format("read segment index at %d", p)
end
sip = getLong(buff, 1)
if ip < sip then
h = m - 1
else
eip = getLong(buff, 5)
if ip > eip then
l = m + 1
else
data_len = getShort(buff, 9)
data_ptr = getLong(buff, 11)
break
end
end
end
-- matching nothing interception
-- print(string.format("data_len=%d, data_ptr=%d", data_len, data_ptr))
if data_len == 0 then
return "", nil
end
-- load and return the region data
buff, err = read_data(self, data_ptr, data_len)
if err ~= nil then
return "", string.format("read data at %d:%d", data_ptr, data_len)
end
return buff, nil
end
-- read specified bytes from the specified index
function _M:read(offset, length)
-- local cache
local content_buff = self.content_buff
local handle = self.handle
-- check the in-memory buffer first
if content_buff ~= nil then
return string.sub(content_buff, offset + 1, offset + length), nil
end
-- read from the file
local r = handle:seek("set", offset)
if r == nil then
return nil, string.format("seek to offset %d", offset)
end
self.io_count = self.io_count + 1
local buff = handle:read(length)
if buff == nil then
return nil, string.format("read %d bytes", length)
end
return buff, nil
end
function _M:get_io_count()
return self.io_count
end
function _M:close()
if self.handle ~= nil then
self.handle:close()
end
end
-- End of search api
-- static util functions
function _M.load_header(dbPath)
local handle = io.open(dbPath, "r")
if handle == nil then
return nil, string.format("failed to open xdb file `%s`", dbPath)
end
local r = handle:seek("set", 0)
if r == nil then
handle:close()
return nil, "failed to seek to 0"
end
local c = handle:read(HeaderInfoLength)
if c == nil then
handle:close()
return nil, string.format("failed to read %d bytes", HeaderInfoLength)
end
handle:close()
return {
["version"] = getShort(c, 1),
["index_policy"] = getShort(c, 3),
["created_at"] = getLong(c, 5),
["start_index_ptr"] = getLong(c, 9),
["end_index_ptr"] = getLong(c, 13),
["raw_data"] = c
}, nil
end
function _M.load_vector_index(dbPath)
local handle = io.open(dbPath, "r")
if handle == nil then
return nil, string.format("failed to open xdb file `%s`", dbPath)
end
local r = handle:seek("set", HeaderInfoLength)
if r == nil then
handle:close()
return nil, string.format("failed to seek to %d", HeaderInfoLength)
end
local c = handle:read(VectorIndexLength)
if c == nil then
handle:close()
return nil, string.format("failed to read %d bytes", VectorIndexLength)
end
handle:close()
return c, nil
end
function _M.load_content(dbPath)
local handle = io.open(dbPath, "r")
if handle == nil then
return nil, string.format("failed to open xdb file `%s`", dbPath)
end
local c = handle:read("*a")
if c == nil then
handle:close()
return nil, string.format("failed to read xdb content")
end
handle:close()
return c, nil
end
function _M.check_ip(ip_str)
local ip, id, v = 0, 1, 0
local offset_arr = {24, 16, 8, 0}
for p in string.gmatch(ip_str..".", "([%d]+)%.") do
-- match pattern checking
if p == nil then
return 0, "err=1"
end
-- count checking
if id > 4 then
return 0, "err=1"
end
-- value checking
v = tonumber(p)
if v > 255 then
return 0, "err=2"
end
ip = ip | (v << offset_arr[id])
id = id + 1
end
if id ~= 5 then
return 0, "err=1"
end
return ip, nil
end
function _M.long2ip(ip)
return string.format("%d.%d.%d.%d", (ip >> 24) & 0xFF, (ip >> 16) & 0xFF, (ip >> 8 ) & 0xFF, ip & 0xFF)
end
-- this is a bit weird, but we have to better choice for now
function _M.now()
return os.time() * 1e6
end
-- End of util functions
--internal function to get a integer from a binary string
function getLong(buff, idx)
local i1 = (string.byte(string.sub(buff, idx, idx)))
local i2 = (string.byte(string.sub(buff, idx+1, idx+1)) << 8)
local i3 = (string.byte(string.sub(buff, idx+2, idx+2)) << 16)
local i4 = (string.byte(string.sub(buff, idx+3, idx+3)) << 24)
return (i1 | i2 | i3 | i4)
end
function getShort(buff, idx)
local i1 = (string.byte(string.sub(buff, idx, idx)))
local i2 = (string.byte(string.sub(buff, idx+1, idx+1)) << 8)
return (i1 | i2)
end
return _M

16
binding/lua_c/Makefile
View File

@ -1,16 +0,0 @@
LuaVersion = 5.4
LIB_DIR = /usr/local/share/lua/$(LuaVersion)
all: ../c/xdb_api.h ../c/xdb_util.c ../c/xdb_searcher.c xdb_searcher.c
gcc -g -O2 -I../c/ -I/usr/include/lua$(LuaVersion) ../c/xdb_util.c ../c/xdb_searcher.c xdb_searcher.c -fPIC -shared -o xdb_searcher.so
install:
sudo mkdir -p $(LIB_DIR); \
sudo cp xdb_searcher.so $(LIB_DIR);\
echo "install xdb searcher to $(LIB_DIR) successfully.";\
clean:
find . -name \*.so | xargs rm -f
find . -name \*.o | xargs rm -f
.PHONY: clean

235
binding/lua_c/ReadMe.md
View File

@ -1,235 +0,0 @@
# ip2region xdb lua c 扩展查询客户端实现
# 编译安装
通过如下方式来编译安装该扩展：
```bash
# cd 到 lua_c binding 的根目录
make
sudo make install
```
备注：Makefile 里面的 `LuaVersion` 变量代表你本地环境的 lua 的版本，默认为 5.4，如果你的版本不是 5.4，记得先修改为和你本地 lua 版本一致的版本号。
# 使用方式
### 关于 IPv4 和 IPv6
该 xdb 查询客户端实现同时支持对 IPv4 和 IPv6 的查询，使用方式如下：
```lua
-- 引入 xdb searcher 扩展
local xdb = require("xdb_searcher")
-- 如果是 IPv4: 设置 xdb 路径为 v4 的 xdb 文件，IP版本指定为 IPv4
local db_path = "../../data/ip2region_v4.xdb" -- 或者你的 ipv4 xdb 的路径
local version = xdb.IPv4
-- 如果是 IPv6: 设置 xdb 路径为 v6 的 xdb 文件，IP版本指定为 IPv6
local db_path = "../../data/ip2region_v6.xdb"; -- 或者你的 ipv6 xdb 路径
local version = xdb.IPv6
-- db_path 指定的 xdb 的 IP 版本必须和 version 指定的一致，不然查询执行的时候会报错
-- 备注：以下演示直接使用 db_path 和 version 变量
```
### XDB 文件验证
建议您主动去验证 xdb 文件的适用性，因为后期的一些新功能可能会导致目前的 Searcher 版本无法适用你使用的 xdb 文件，验证可以避免运行过程中的一些不可预测的错误。 你不需要每次都去验证，例如在服务启动的时候，或者手动调用命令验证确认版本匹配即可，不要在每次创建的 Searcher 的时候运行验证，这样会影响查询的响应速度，尤其是高并发的使用场景。
```lua
local xdb = require("xdb_searcher")
-- verify the xdb
if xdb.verify(db_path) == false then
-- 适用性验证失败！！！
-- 当前查询客户端实现不适用于 db_path 指定的 xdb 文件的查询.
-- 应该停止启动服务，使用合适的 xdb 文件或者升级到适合 db_path 的 Searcher 实现。
print(string.format("failed to verify the xdb file: %s", db_path))
return
end
-- 验证通过，当前使用的 Searcher 可以安全的用于对 db_path 指向的 xdb 的查询操作
```
### 完全基于文件的查询
```lua
local xdb = require("xdb_searcher")
-- 1、使用 version 从 db_path 创建基于文件的 xdb 查询对象
local searcher, err = xdb.new_with_file_only(version, db_path)
if err ~= nil then
print(string.format("failed to create searcher: %s", err))
return
end
-- 2、调用查询 API 进行查询，IPv4 和 IPv6 都支持
local ip_str = "1.2.3.4"
-- ip_str = "2001:4:112:ffff:ffff:ffff:ffff:ffff" // IPv6
local s_time = xdb.now()
region, err = searcher:search(ip_str)
local c_time = xdb.now() - s_time
if err ~= nil then
print(string.format("failed to search(%s): %s", ip_str, err))
return
end
print(string.format("{region: %s, took: %.5f μs}", region, c_time))
-- 备注：并发使用，每个协程需要创建单独的 xdb 查询对象
-- 3，关闭 xdb 查询器
searcher:close()
--
-- 4，模块资源清理，仅在需要将整个服务完全关闭前调用
xdb.cleanup()
```
### 缓存 `VectorIndex` 索引
如果你的 `lua` 母环境支持，可以预先加载 vectorIndex 缓存，然后做成全局变量，每次创建 Searcher 的时候使用全局的 vectorIndex，可以减少一次固定的 IO 操作从而加速查询，减少 io 压力。
```lua
local xdb = require("xdb_searcher")
-- 1、从指定的 db_path 加载 VectorIndex 缓存，把下述的 v_index 对象做成全局变量。
-- vectorIndex 加载一次即可，建议在服务启动的时候加载为全局对象。
v_index, err = xdb.load_vector_index(db_path)
if err ~= nil then
print(string.format("failed to load vector index from '%s'", db_path))
return
end
-- 2、使用全局的 v_index 创建带 VectorIndex 缓存的查询对象。
searcher, err = xdb.new_with_vector_index(version, db_path, v_index)
if err ~= nil then
print(string.format("failed to create vector index searcher: %s", err))
return
end
-- 3、调用查询 API ，IPv4 和 IPv6 都支持
local ip_str = "1.2.3.4"
-- ip_str = "2001:4:112:ffff:ffff:ffff:ffff:ffff" // IPv6
local s_time = xdb.now()
region, err = searcher:search(ip_str)
local c_time = xdb.now() = s_time
if err ~= nil then
print(string.format("failed to search(%s): %s", ip_str, err))
return
end
print(string.format("{region: %s, took: %.5f μs}", region, c_time))
-- 备注：并发使用，每个协程需要创建单独的 xdb 查询对象，但是共享全局的 v_index 对象
-- 4，关闭 xdb 查询器
searcher:close()
--
-- 5，模块资源清理，仅在需要将整个服务完全关闭前调用
xdb.cleanup()
```
### 缓存整个 `xdb` 数据
如果你的 `lua` 母环境支持，可以预先加载整个 xdb 的数据到内存，这样可以实现完全基于内存的查询，类似之前的 memory search 查询。
```lua
local xdb = require("xdb_searcher")
-- 1、从指定的 db_path 加载整个 xdb 到内存。
-- xdb内容加载一次即可，建议在服务启动的时候加载为全局对象。
local content = xdb.load_content(db_path)
if content == nil then
print(string.format("failed to load xdb content from '%s'", db_path))
return
end
-- 2、使用全局的 content 创建带完全基于内存的查询对象。
searcher, err = xdb.new_with_buffer(version, content)
if err ~= nil then
print(string.format("failed to create content buffer searcher: %s", err))
return
end
-- 3、调用查询 API ，IPv4 和 IPv6 都支持
local ip_str = "1.2.3.4"
-- ip_str = "2001:4:112:ffff:ffff:ffff:ffff:ffff" // IPv6
local s_time = xdb.now()
region, err = searcher:search(ip_str)
local c_time = xdb.now() - s_time
if err ~= nil then
print(string.format("failed to search(%s): %s", ip_str, err))
return
end
print(string.format("{region: %s, took: %.5f μs}", region, c_time))
-- 备注：并发使用，用 xdb 整个缓存创建的查询对象可以安全的用于并发。
-- 建议在服务启动的时候创建好全局的 searcher 对象，然后全局并发使用。
-- 4，关闭 xdb 查询器
searcher:close()
--
-- 5，模块资源清理，仅在需要将整个服务完全关闭前调用
xdb.cleanup()
```
# 查询测试
通过 `search_test.lua` 脚本来进行查询测试：
```bash
➜ lua_c git:(fr_lua_c_ipv6) ✗ lua ./search_test.lua
lua search_test.lua [command options]
options:
--db string ip2region binary xdb file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：使用默认的 data/ip2region_v4.xdb 进行 IPv4 查询测试：
```bash
➜ lua_c git:(fr_lua_c_ipv6) ✗ lua ./search_test.lua --db=../../data/ip2region_v4.xdb
ip2region xdb searcher test program
source xdb: ../../data/ip2region_v4.xdb (IPv4, vectorIndex)
type 'quit' to exit
ip2region>> 120.229.45.2
{region: 中国|广东省|深圳市|移动, io_count: 3, took: 34μs}
```
例如：使用默认的 data/ip2region_v6.xdb 进行 IPv6 查询测试：
```bash
➜ lua_c git:(fr_lua_c_ipv6) ✗ lua ./search_test.lua --db=../../data/ip2region_v6.xdb
ip2region xdb searcher test program
source xdb: ../../data/ip2region_v6.xdb (IPv6, vectorIndex)
type 'quit' to exit
ip2region>> 240e:3b7:3276:33b0:958f:f34c:d04f:f6a
{region: 中国|广东省|深圳市|家庭宽带, io_count: 14, took: 79μs}
```
输入 ip 即可进行查询测试。也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的效率。
# bench 测试
通过 `bench_test.lua` 脚本来进行自动 bench 测试，一方面确保 `xdb` 文件没有错误，另一方面通过大量的查询测试平均查询性能：
```bash
➜ lua_c git:(fr_lua_c_ipv6) ✗ lua ./bench_test.lua
lua bench_test.lua [command options]
options:
--db string ip2region binary xdb file path
--src string source ip text file path
--cache-policy string cache policy: file/vectorIndex/content
```
例如：通过默认的 data/ip2region_v4.xdb 和 data/ipv4_source.txt 来进行 IPv4 的 bench 测试：
```bash
➜ lua_c git:(fr_lua_c_ipv6) ✗ lua ./bench_test.lua --db=../../data/ip2region_v4.xdb --src=../../data/ipv4_source.txt
Bench finished, {cachePolicy: vectorIndex, total: 1367686, took: 8.593 s, cost: 5.433 μs/op}
```
例如：通过默认的 data/ip2region_v6.xdb 和 data/ipv6_source.txt 来进行 IPv6 的 bench 测试：
```bash
➜ lua_c git:(fr_lua_c_ipv6) ✗ lua ./bench_test.lua --db=../../data/ip2region_v6.xdb --src=../../data/ipv6_source.txt
Bench finished, {cachePolicy: vectorIndex, total: 34159862, took: 829.008 s, cost: 23.176 μs/op}
```
可以通过设置 `cache-policy` 参数来分别测试 file/vectorIndex/content 三种不同的缓存实现的的性能。
@Note：请注意 bench 使用的 src 文件需要是生成对应的 xdb 文件的相同的源文件。

201
binding/lua_c/bench_test.lua
View File

@ -1,201 +0,0 @@
-- Copyright 2022 The Ip2Region Authors. All rights reserved.
-- Use of this source code is governed by a Apache2.0-style
-- license that can be found in the LICENSE file.
--
-- ---
-- @Author Lion <chenxin619315@gmail.com>
-- @Date 2022/06/30
-- set the package to load the current xdb_searcher.so
package.path = "./?.lua" .. package.path
package.cpath = "./?.so" .. package.cpath
local xdb = require("xdb_searcher")
function printHelp()
print("lua bench_test.lua [command options]")
print("options: ")
print(" --db string ip2region binary xdb file path")
print(" --src string source ip text file path")
print(" --cache-policy string cache policy: file/vectorIndex/content")
end
if #arg < 2 then
printHelp(arg)
return
end
-- parser the command line args
local dbFile, srcFile = "", ""
local cachePolicy = "vectorIndex"
for _, r in ipairs(arg) do
if string.len(r) < 5 then
goto continue
end
if string.sub(r, 1, 2) ~= "--" then
goto continue
end
for k, v in string.gmatch(string.sub(r, 3), "([^=]+)=([^%s]+)") do
if k == "db" then
dbFile = v
elseif k == "src" then
srcFile = v
elseif k == "cache-policy" then
cachePolicy = v
else
print(string.format("undefined option `%s`", r))
return
end
-- break the match iterate
break
end
-- continue this loop
::continue::
end
-- print(string.format("dbFile=%s, srcFile=%s, cachePolicy=%s", dbFile, srcFile, cachePolicy))
if string.len(dbFile) < 2 or string.len(srcFile) < 2 then
printHelp()
return
end
-- verify the xdb from header
if xdb.verify(dbFile) == false then
print(string.format("failed to verify the xdb file: %s", dbFile))
return
end
-- detect the version from the xdb header
header, err = xdb.load_header(dbFile)
if err ~= nil then
print(string.format("failed to load header: %s", err))
return
end
version, err = xdb.version_from_header(header);
if err ~= nil then
print(string.format("failed to detect version from header: %s", err))
return
end
-- create the searcher based on the cache-policy
local searcher, v_index, content
if cachePolicy == "file" then
searcher, err = xdb.new_with_file_only(version, dbFile)
if err ~= nil then
print(string.format("failed to create searcher: %s", err))
return
end
elseif cachePolicy == "vectorIndex" then
v_index, err = xdb.load_vector_index(dbFile)
if err ~= nil then
print(string.format("failed to load vector index: %s", err))
return
end
searcher, err = xdb.new_with_vector_index(version, dbFile, v_index)
if err ~= nil then
print(string.format("failed to create vector index searcher: %s", err))
return
end
elseif cachePolicy == "content" then
content, err = xdb.load_content(dbFile)
if err ~= nil then
print(string.format("failed to load xdb content from '%s'", dbFile))
return
end
searcher, err = xdb.new_with_buffer(version, content)
if err ~= nil then
print(string.format("failed to create content buffer searcher: %s", err))
return
end
else
print(string.format("undefined cache-policy `%s`", cachePolicy))
return
end
-- do the bench test
local handle = io.open(srcFile, "r")
if handle == nil then
print(string.format("failed to open src text file `%s`", handle))
return
end
local lines = handle:lines()
local sip_str, eip_str, s_region, region = "", "", "", ""
local sip, mip, eip, err = 0, 0, 0, 0
local count, t_time, c_time = 0, 0, 0
local s_time = xdb.now()
for l in lines do
if string.len(l) < 1 then
goto continue
end
for v1, v2, v3 in string.gmatch(l, "([^|]+)|([^|]+)|([^\n]+)") do
-- print(sip_str, eip_str, region)
sip_str = v1
eip_str = v2
s_region = v3
break
end
sip_bytes, err = xdb.parse_ip(sip_str)
if err ~= nil then
print(string.format("invalid start ip `%s`", sip_str))
return
end
eip_bytes, err = xdb.parse_ip(eip_str)
if err ~= nil then
print(string.format("invalid end ip `%s`", sip_str))
return
end
if xdb.ip_compare(sip_bytes, eip_bytes) > 0 then
print(string.format("start ip(%s) should not be greater than end ip(%s)\n", sip_str, eip_str))
return
end
for _, ip_bytes in ipairs({sip_bytes, eip_bytes}) do
t_time = xdb.now()
region, err = searcher:search(ip_bytes)
c_time = c_time + xdb.now() - t_time
if err ~= nil then
print(string.format("failed to search ip `%s`", xdb.ip_to_string(ip_bytes)))
return
end
-- check the region
if region ~= s_region then
printf(string.format("failed search(%s) with (%s != %s)\n", xdb.ip_to_string(ip_bytes), region, s_region))
return
end
count = count + 1
end
::continue::
end
-- resource cleanup
searcher:close()
if v_index ~= nil then
v_index:close()
end
if content ~= nil then
content:close()
end
xdb.cleanup();
-- print the stats
local avg_costs = 0
if count > 0 then
avg_costs = c_time / count
end
print(string.format("Bench finished, {cachePolicy: %s, total: %d, took: %.3f s, cost: %.3f μs/op}",
cachePolicy, count, (xdb.now() - s_time)/1e6, c_time / count))

172
binding/lua_c/search_test.lua
View File

@ -1,172 +0,0 @@
-- Copyright 2022 The Ip2Region Authors. All rights reserved.
-- Use of this source code is governed by a Apache2.0-style
-- license that can be found in the LICENSE file.
--
-- ---
-- @Author Lion <chenxin619315@gmail.com>
-- @Date 2022/06/30
-- set the package to load the current xdb_searcher.so
package.path = "./?.lua" .. package.path
package.cpath = "./?.so" .. package.cpath
local xdb = require("xdb_searcher")
function printHelp()
print("lua search_test.lua [command options]")
print("options: ")
print(" --db string ip2region binary xdb file path")
print(" --cache-policy string cache policy: file/vectorIndex/content")
end
if #arg < 1 then
printHelp(arg)
return
end
-- parser the command line args
local dbFile = ""
local cachePolicy = "vectorIndex"
for _, r in ipairs(arg) do
if string.len(r) < 5 then
goto continue
end
if string.sub(r, 1, 2) ~= "--" then
goto continue
end
for k, v in string.gmatch(string.sub(r, 3), "([^=]+)=([^%s]+)") do
if k == "db" then
dbFile = v
elseif k == "cache-policy" then
cachePolicy = v
else
print(string.format("undefined option `%s`", r))
return
end
-- break the match iterate
break
end
-- continue this loop
::continue::
end
-- print(string.format("dbFile=%s, cachePolicy=%s", dbFile, cachePolicy))
if string.len(dbFile) < 2 then
printHelp()
return
end
-- verify the xdb
if xdb.verify(dbFile) == false then
print(string.format("failed to verify the xdb file: %s", dbFile))
return
end
-- detect the version from the xdb header
header, err = xdb.load_header(dbFile)
if err ~= nil then
print(string.format("failed to load header: %s", err))
return
end
version, err = xdb.version_from_header(header);
if err ~= nil then
print(string.format("failed to detect version from header: %s", err))
return
end
-- create the searcher based on the cache-policy
local searcher, v_index, content
if cachePolicy == "file" then
searcher, err = xdb.new_with_file_only(version, dbFile)
if err ~= nil then
print(string.format("failed to create searcher: %s", err))
return
end
elseif cachePolicy == "vectorIndex" then
v_index, err = xdb.load_vector_index(dbFile)
if err ~= nil then
print(string.format("failed to load vector index: %s", err))
return
end
searcher, err = xdb.new_with_vector_index(version, dbFile, v_index)
if err ~= nil then
print(string.format("failed to create vector index searcher: %s", err))
return
end
elseif cachePolicy == "content" then
content, err = xdb.load_content(dbFile)
if err ~= nil then
print(string.format("failed to load xdb content from '%s'", dbFile))
return
end
searcher, err = xdb.new_with_buffer(version, content)
if err ~= nil then
print(string.format("failed to create content buffer searcher: %s", err))
return
end
else
print(string.format("undefined cache-policy `%s`", cachePolicy))
return
end
-- do the search
print(string.format([[
ip2region xdb searcher test program
source xdb: %s (%s, %s)
type 'quit' to exit]], dbFile, xdb.version_info(version).name, cachePolicy))
local region, err = "", nil
local ip_int, s_time, c_time = 0, 0, 0
while ( true ) do
io.write("ip2region>> ");
io.input(io.stdin);
local line = io.read();
if (line == nil) then
break
end
if ( line == "quit" ) then
break
end
-- empty string ignore
line = line:gsub("^%s*(.-)%s*$", "%1")
if string.len(line) < 1 then
goto continue
end
ip_bytes, err = xdb.parse_ip(line)
-- print(string.format("parse(%s): %s, err: %s", line, xdb.ip_to_string(ip_bytes), err))
if err ~= nil then
print(string.format("invalid ip address `%s`", line))
goto continue
end
-- do the search
s_time = xdb.now()
region, err = searcher:search(ip_bytes)
c_time = xdb.now() - s_time
if err ~= nil then
print(string.format("{err: %s, io_count: %d}", err, searcher:get_io_count()))
else
print(string.format("{region: %s, io_count: %d, took: %dμs}", region, searcher:get_io_count(), c_time))
end
::continue::
end
-- resource cleanup
searcher:close()
if v_index ~= nil then
v_index:close()
end
if content ~= nil then
content:close()
end
xdb.cleanup();

140
binding/lua_c/util_test.lua
View File

@ -1,140 +0,0 @@
-- Copyright 2022 The Ip2Region Authors. All rights reserved.
-- Use of this source code is governed by a Apache2.0-style
-- license that can be found in the LICENSE file.
--
-- ---
-- @Author Lion <chenxin619315@gmail.com>
-- @Date 2022/06/30
-- set the package to load the current xdb_searcher.so
package.path = "./?.lua" .. package.path
package.cpath = "./?.so" .. package.cpath
local xdb = require("xdb_searcher")
---- ip checking testing
function test_parse_ip()
local ip_list = {
"1.2.3.4", "192.168.2.3", "120.24.78.129", "255.255.255.0", "invalid-ipv.4",
"::", "3000::", "240e:3b7:3276:33b0:4844:6f28:f69c:1eee", "2001:4:112::", "invalid-ipv::6"
}
local s_time = xdb.now()
for _, ip_src in ipairs(ip_list) do
ip_bytes, err = xdb.parse_ip(ip_src)
if err ~= nil then
print(string.format("invalid ip address `%s`: %s", ip_src, err))
else
local ip_string = xdb.ip_to_string(ip_bytes);
print(string.format("parse_ip(%s)->%s ? %s", ip_src, ip_string, ip_src==ip_string))
end
end
end
function test_print_const()
print("ipv4: ", xdb.IPv4);
print("ipv6: ", xdb.IPv6);
print("header_buffer: ", xdb.header_buffer);
print("v_index_buffer: ", xdb.v_index_buffer);
print("content_buffer: ", xdb.content_buffer);
end
---- buffer loading test
function test_load_header()
header, err = xdb.load_header("../../data/ip2region_v4.xdb")
if err ~= nil then
print("failed to load header: ", err)
else
print(string.format("xdb header buffer `%s` loaded", header))
local tpl = [[
header: {
version: %d
index_policy: %d
created_at: %d
start_index_ptr: %d
end_index_ptr: %d
ip_version: %d
runtime_ptr_bytes: %d
}]]
local t = header:to_table()
print(string.format(tpl,
t["version"], t["index_policy"], t["created_at"],
t["start_index_ptr"], t["end_index_ptr"], t["ip_version"], t["runtime_ptr_bytes"])
)
end
end
function test_version_info()
local v4 = xdb.version_info(xdb.IPv4)
print(string.format("{id:%d, name: %s, bytes: %d, segment_index_size: %d}", v4.id, v4.name, v4.bytes, v4.segment_index_size))
local v6 = xdb.version_info(xdb.IPv6)
print(string.format("{id:%d, name: %s, bytes: %d, segment_index_size: %d}", v6.id, v6.name, v6.bytes, v6.segment_index_size))
local vx = xdb.version_info(3)
end
function test_load_vector_index()
v_index, err = xdb.load_vector_index("../../data/ip2region_v4.xdb")
if err ~= nil then
print("failed to load vector index: ", err)
else
print(string.format("xdb vector index buffer `%s` loaded, info={name=%s, type=%d, length=%d}",
v_index, v_index:name(), v_index:type(), v_index:length()))
v_index:close()
end
end
function test_load_content()
c_buffer, err = xdb.load_content("../../data/ip2region_v4.xdb")
if err ~= nil then
print("failed to load content: ", err)
else
print(string.format("xdb content buffer `%s` loaded, info={name=%s, type=%d, length=%d}",
c_buffer, c_buffer:name(), c_buffer:type(), c_buffer:length()))
c_buffer:close();
end
end
function test_search()
-- ipv4
local ip_str = "1.2.3.4"
searcher, err = xdb.new_with_file_only(xdb.IPv4, "../../data/ip2region_v4.xdb")
print(string.format("searcher.tostring=%s", searcher))
local t_start = xdb.now()
region, err = searcher:search(ip_str)
local c_time = xdb.now() - t_start
print(string.format("search(%s): {region=%s, io_count: %d, took: %dμs, err=%s}",
ip_str, region, searcher:get_io_count(), c_time, err))
searcher:close()
-- IPv6
ip_str = "240e:3b7:3276:33b0:958f:f34c:d04f:f6a"
searcher, err = xdb.new_with_file_only(xdb.IPv6, "../../data/ip2region_v6.xdb")
print(string.format("searcher.tostring=%s", searcher))
t_start = xdb.now()
region, err = searcher:search(ip_str)
c_time = xdb.now() - t_start
print(string.format("search(%s): {region=%s, io_count: %d, took: %dμs, err=%s}",
ip_str, region, searcher:get_io_count(), c_time, err))
searcher:close()
end
local func_name = arg[1]
if func_name == nil then
print("please specified the function to test")
return
end
if (_G[func_name] == nil) then
print(string.format("undefined function `%s` to call", func_name))
return
end
local s_time = xdb.now();
print(string.format("+---calling test function %s ...", func_name))
_G[func_name]();
local cost_time = xdb.now() - s_time
xdb.cleanup();
print(string.format("|---done, took: %.3fμs", cost_time))

761
binding/lua_c/xdb_searcher.c
View File

@ -1,761 +0,0 @@
// Copyright 2022 The Ip2Region Authors. All rights reserved.
// Use of this source code is governed by a Apache2.0-style
// license that can be found in the LICENSE file.
// ---
// @Author Lion <chenxin619315@gmail.com>
// @Date 2022/06/30
#include "stdio.h"
#include "lua.h"
#include "lauxlib.h"
#include "../c/xdb_api.h"
#define XDB_BUFFER_METATABLE_NAME "xdb_buffer_mt"
#define XDB_METATABLE_NAME "xdb_metatable_name"
#define xdb_header_buffer 1
#define xdb_vector_index_buffer 2
#define xdb_content_buffer 3
// --- xdb buffer interface impl
struct xdb_buffer_entry {
int type; // buffer type
char *name; // buffer name
void *ptr; // buffer ptr
void (*closer) (void *);
};
typedef struct xdb_buffer_entry xdb_buffer_t;
static int lua_xdb_buffer_name(lua_State *L) {
xdb_buffer_t *buffer;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via ':'");
buffer = (xdb_buffer_t *) luaL_checkudata(L, 1, XDB_BUFFER_METATABLE_NAME);
lua_pushstring(L, buffer->name);
return 1;
}
static int lua_xdb_buffer_type(lua_State *L) {
xdb_buffer_t *buffer;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via ':'");
buffer = (xdb_buffer_t *) luaL_checkudata(L, 1, XDB_BUFFER_METATABLE_NAME);
lua_pushinteger(L, buffer->type);
return 1;
}
static int lua_xdb_buffer_to_table(lua_State *L) {
xdb_buffer_t *buffer;
xdb_header_t *header;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via ':'");
buffer = (xdb_buffer_t *) luaL_checkudata(L, 1, XDB_BUFFER_METATABLE_NAME);
lua_newtable(L);
if (buffer->type == xdb_header_buffer) {
header = (xdb_header_t *) buffer->ptr;
lua_pushinteger(L, header->version);
lua_setfield(L, -2, "version");
lua_pushinteger(L, header->index_policy);
lua_setfield(L, -2, "index_policy");
lua_pushinteger(L, header->created_at);
lua_setfield(L, -2, "created_at");
lua_pushinteger(L, header->start_index_ptr);
lua_setfield(L, -2, "start_index_ptr");
lua_pushinteger(L, header->end_index_ptr);
lua_setfield(L, -2, "end_index_ptr");
lua_pushinteger(L, header->ip_version);
lua_setfield(L, -2, "ip_version");
lua_pushinteger(L, header->runtime_ptr_bytes);
lua_setfield(L, -2, "runtime_ptr_bytes");
} else {
// do nothing for now
}
return 1;
}
static int lua_xdb_buffer_length(lua_State *L) {
xdb_buffer_t *buffer;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via ':'");
buffer = (xdb_buffer_t *) luaL_checkudata(L, 1, XDB_BUFFER_METATABLE_NAME);
if (buffer->type == xdb_header_buffer) {
lua_pushinteger(L, ((xdb_header_t *) buffer->ptr)->length);
} else if (buffer->type == xdb_vector_index_buffer) {
lua_pushinteger(L, ((xdb_vector_index_t *) buffer->ptr)->length);
} else if (buffer->type == xdb_content_buffer) {
lua_pushinteger(L, ((xdb_content_t *) buffer->ptr)->length);
} else {
lua_pushinteger(L, -1);
}
return 1;
}
static int lua_xdb_buffer_tostring(lua_State *L) {
xdb_buffer_t *buffer;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via ':'");
buffer = (xdb_buffer_t *) luaL_checkudata(L, 1, XDB_BUFFER_METATABLE_NAME);
lua_pushfstring(L, "xdb %s buffer object {name: %s, type: %d}", buffer->name, buffer->name, buffer->type);
return 1;
}
static int lua_xdb_buffer_close(lua_State *L) {
xdb_buffer_t *buffer;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via ':'");
buffer = (xdb_buffer_t *) luaL_checkudata(L, 1, XDB_BUFFER_METATABLE_NAME);
// check and call the closer
if (buffer->closer != NULL) {
buffer->closer(buffer->ptr);
buffer->closer = NULL;
}
return 0;
}
// module method define, should be access via ':'
static const struct luaL_Reg xdb_buffer_methods[] = {
{"name", lua_xdb_buffer_name},
{"type", lua_xdb_buffer_type},
{"length", lua_xdb_buffer_length},
{"to_table", lua_xdb_buffer_to_table},
{"close", lua_xdb_buffer_close},
{"__gc", lua_xdb_buffer_close},
{"__tostring", lua_xdb_buffer_tostring},
{NULL, NULL}
};
// --- End of xdb buffer
// --- xdb util function
static int lua_xdb_load_header_from_file(lua_State *L) {
const char *db_path;
xdb_header_t *header;
xdb_buffer_t *buffer;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via '.' and the xdb file path expected");
db_path = luaL_checkstring(L, 1);
header = xdb_load_header_from_file(db_path);
if (header == NULL) {
lua_pushnil(L);
lua_pushfstring(L, "load header from `%s`", db_path);
return 2;
}
// alloc the buffer.
buffer = (xdb_buffer_t *) lua_newuserdata(L, sizeof(xdb_buffer_t));
if (buffer == NULL) {
lua_pushnil(L);
lua_pushfstring(L, "failed to alloc xdb buffer entry");
return 2;
}
// init the buffer
buffer->type = xdb_header_buffer;
buffer->name = "header";
buffer->ptr = header;
buffer->closer = xdb_free_header;
// set the metatable of the header buffer object and push onto the stack
luaL_getmetatable(L, XDB_BUFFER_METATABLE_NAME);
lua_setmetatable(L, -2);
lua_pushnil(L);
return 2;
}
static int lua_xdb_load_vector_index_from_file(lua_State *L) {
const char *db_path;
xdb_vector_index_t *v_index;
xdb_buffer_t *buffer;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via '.' and the xdb path expected");
db_path = luaL_checkstring(L, 1);
v_index = xdb_load_vector_index_from_file(db_path);
if (v_index == NULL) {
lua_pushnil(L);
lua_pushfstring(L, "load vector index from `%s`", db_path);
return 2;
}
// alloc the buffer.
buffer = (xdb_buffer_t *) lua_newuserdata(L, sizeof(xdb_buffer_t));
if (buffer == NULL) {
lua_pushnil(L);
lua_pushstring(L, "failed to alloc xdb buffer entry");
return 2;
}
// init the buffer
buffer->type = xdb_vector_index_buffer;
buffer->name = "v_index";
buffer->ptr = v_index;
buffer->closer = xdb_free_vector_index;
// set the metatable of the header buffer object and push onto the stack
luaL_getmetatable(L, XDB_BUFFER_METATABLE_NAME);
lua_setmetatable(L, -2);
lua_pushnil(L);
return 2;
}
static int lua_xdb_load_content_from_file(lua_State *L) {
const char *db_path;
xdb_content_t *content;
xdb_buffer_t *buffer;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via '.' and xdb path expected");
db_path = luaL_checkstring(L, 1);
content = xdb_load_content_from_file(db_path);
if (content == NULL) {
lua_pushnil(L);
lua_pushfstring(L, "load xdb content from `%s`", db_path);
return 2;
}
// alloc the buffer.
buffer = (xdb_buffer_t *) lua_newuserdata(L, sizeof(xdb_buffer_t));
if (buffer == NULL) {
lua_pushnil(L);
lua_pushstring(L, "failed to alloc xdb buffer entry");
return 2;
}
// init the buffer
buffer->type = xdb_content_buffer;
buffer->name = "content";
buffer->ptr = content;
buffer->closer = xdb_free_content;
// set the metatable of the header buffer object and push onto the stack
luaL_getmetatable(L, XDB_BUFFER_METATABLE_NAME);
lua_setmetatable(L, -2);
lua_pushnil(L);
return 2;
}
static int lua_xdb_verify_from_file(lua_State *L) {
const char *db_path;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via '.' and the xdb file path expected");
db_path = luaL_checkstring(L, 1);
lua_pushboolean(L, xdb_verify_from_file(db_path) == 0 ? 1 : 0);
return 1;
}
static int lua_xdb_version_from_header(lua_State *L) {
xdb_version_t *version;
xdb_buffer_t *buffer;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via '.' and xdb header expected");
// header buffer checking
buffer = luaL_checkudata(L, 1, XDB_BUFFER_METATABLE_NAME);
if (buffer->type != xdb_header_buffer) {
return luaL_error(L, "invalid xdb header buffer");
}
version = xdb_version_from_header((xdb_header_t *) buffer->ptr);
if (version == NULL) {
lua_pushnil(L);
lua_pushstring(L, "failed to detect version from header");
} else {
lua_pushinteger(L, version->id);
lua_pushnil(L);
}
return 2;
}
static xdb_version_t *_get_version(lua_State *L, int arg) {
int vid = luaL_checkinteger(L, arg);
if (vid == xdb_ipv4_id) {
return XDB_IPv4;
} else if (vid == xdb_ipv6_id) {
return XDB_IPv6;
} else {
return NULL;
}
}
static int lua_xdb_version_info(lua_State *L) {
xdb_version_t *version;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via '.' and version id expected");
// check the ip version
version = _get_version(L, 1);
if (version == NULL) {
return luaL_error(L, "invalid verison id specified");
}
lua_newtable(L);
lua_pushinteger(L, version->id);
lua_setfield(L, -2, "id");
lua_pushstring(L, version->name);
lua_setfield(L, -2, "name");
lua_pushinteger(L, version->bytes);
lua_setfield(L, -2, "bytes");
lua_pushinteger(L, version->segment_index_size);
lua_setfield(L, -2, "segment_index_size");
return 1;
}
static int lua_xdb_parse_ip(lua_State *L) {
int err;
const char *ip_str;
bytes_ip_t ip_bytes[19] = {'\0'};
xdb_version_t *version;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via '.' and string ip expected, eg: 1.2.3.4 / 3000::");
ip_str = luaL_checkstring(L, 1);
version = xdb_parse_ip(ip_str, ip_bytes + 2, sizeof(ip_bytes) - 2);
if (version == NULL) {
lua_pushnil(L);
lua_pushfstring(L, "failed to parse the `%s`", ip_str);
return 2;
}
// append the magic char for later analysis
// printf("ip:%s, version->id: %d\n", ip_str, version->id);
ip_bytes[0] = '&';
ip_bytes[1] = (bytes_ip_t) version->id;
lua_pushlstring(L, ip_bytes, version->bytes + 2);
lua_pushnil(L);
return 2;
}
static int lua_xdb_ip_to_string(lua_State *L) {
int err, vid, bytes;
const bytes_ip_t *ip_bytes;
char ip_string[INET6_ADDRSTRLEN + 1] = {'\0'};
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via '.' and bytes ip expected");
ip_bytes = luaL_checkstring(L, 1);
if (strlen(ip_bytes) < 2) {
lua_pushnil(L);
lua_pushstring(L, "invalid binary ip bytes specified");
return 2;
}
if (ip_bytes[0] != '&') {
lua_pushnil(L);
lua_pushstring(L, "invalid binary ip bytes specified");
return 2;
}
vid = ip_bytes[1] & 0xFF;
if (vid == xdb_ipv4_id) {
// IPv4
bytes = xdb_ipv4_bytes;
} else if (vid == xdb_ipv6_id) {
// IPv6
bytes = xdb_ipv6_bytes;
} else {
lua_pushnil(L);
lua_pushstring(L, "invalid binary ip bytes specified");
return 2;
}
err = xdb_ip_to_string(ip_bytes + 2, bytes, ip_string, sizeof(ip_string));
if (err != 0) {
lua_pushnil(L);
lua_pushstring(L, "failed to conver the ip bytes to string");
return 2;
}
lua_pushstring(L, ip_string);
lua_pushnil(L);
return 2;
}
static int _validate_bytes_ip(const bytes_ip_t *ip_bytes) {
if (strlen(ip_bytes) < 2) {
return 1;
}
if (ip_bytes[0] != '&') {
return 2;
}
int vid = ip_bytes[1] & 0xFF;
if (vid != xdb_ipv4_id && vid != xdb_ipv6_id) {
return 3;
}
return 0;
}
static int lua_xdb_ip_compare(lua_State *L) {
int err, vid, bytes;
const bytes_ip_t *ip1_bytes, *ip2_bytes;
luaL_argcheck(L, lua_gettop(L) == 2, 1, "call via '.' bytes ip1 and ip2 expected");
ip1_bytes = luaL_checkstring(L, 1);
ip2_bytes = luaL_checkstring(L, 2);
// validate the ip1
err = _validate_bytes_ip(ip1_bytes);
if (err != 0) {
lua_pushnil(L);
lua_pushfstring(L, "failed to validate ip1 with errcode=%d", err);
return 2;
}
// validate the ip2
err = _validate_bytes_ip(ip2_bytes);
if (err != 0) {
lua_pushnil(L);
lua_pushfstring(L, "failed to validate ip2 with errcode=%d", err);
return 2;
}
if (ip1_bytes[1] != ip2_bytes[1]) {
lua_pushnil(L);
lua_pushstring(L, "ip version of ip1 and ip2 are not the same");
return 2;
}
err = xdb_ip_sub_compare(ip1_bytes + 2, (ip1_bytes[1] & 0xFF), ip2_bytes, 2);
lua_pushinteger(L, err);
lua_pushnil(L);
return 2;
}
static int lua_xdb_now(lua_State *L) {
lua_pushinteger(L, xdb_now());
return 1;
}
// --- End of xdb util api
// --- xdb searcher api
static int lua_xdb_new_with_file_only(lua_State *L) {
int err;
xdb_version_t *version;
xdb_searcher_t *searcher;
const char *db_path = NULL;
luaL_argcheck(L, lua_gettop(L) == 2, 1, "call via '.' and ip version / xdb file path expected");
// check the ip version
version = _get_version(L, 1);
if (version == NULL) {
return luaL_error(L, "invalid verison id specified");
}
// check the db path
db_path = luaL_checkstring(L, 2);
// alloc for the searcher
searcher = (xdb_searcher_t *) lua_newuserdata(L, sizeof(xdb_searcher_t));
if (searcher == NULL) {
return luaL_error(L, "failed to alloc xdb searcher entry");
}
// init the xdb searcher
err = xdb_new_with_file_only(version, searcher, db_path);
if (err != 0) {
lua_pushnil(L);
lua_pushfstring(L, "init xdb searcher on `%s`: errcode=%d", db_path, err);
return 2;
}
// push the metatable onto the stack and
// set it as the metatable of the current searcher
luaL_getmetatable(L, XDB_METATABLE_NAME);
lua_setmetatable(L, -2);
lua_pushnil(L);
return 2;
}
static int lua_xdb_new_with_vector_index(lua_State *L) {
xdb_version_t *version;
xdb_searcher_t *searcher;
xdb_buffer_t *xBuffer;
const char *db_path;
int err;
luaL_argcheck(L, lua_gettop(L) == 3, 1, "call via '.', ip version / xdb file path / vector index buffer expected");
// check the ip version
version = _get_version(L, 1);
if (version == NULL) {
return luaL_error(L, "invalid verison id specified");
}
// db_path checking
db_path = luaL_checkstring(L, 2);
// vector index buffer checking
xBuffer = luaL_checkudata(L, 3, XDB_BUFFER_METATABLE_NAME);
if (xBuffer->type != xdb_vector_index_buffer) {
return luaL_error(L, "invalid vector index buffer");
}
// alloc the searcher
searcher = (xdb_searcher_t *) lua_newuserdata(L, sizeof(xdb_searcher_t));
if (searcher == NULL) {
return luaL_error(L, "failed to alloc xdb searcher entry");
}
// init the xdb searcher
err = xdb_new_with_vector_index(version, searcher, db_path, (xdb_vector_index_t *) xBuffer->ptr);
if (err != 0) {
lua_pushnil(L);
lua_pushfstring(L, "init vector index cached xdb searcher on `%s` with errcode=%d", db_path, err);
return 2;
}
// push the metatable onto the stack and
// set it as the metatable of the current searcher
luaL_getmetatable(L, XDB_METATABLE_NAME);
lua_setmetatable(L, -2);
lua_pushnil(L);
return 2;
}
static int lua_xdb_new_with_buffer(lua_State *L) {
xdb_version_t *version;
xdb_searcher_t *searcher;
xdb_buffer_t *xBuffer;
int err;
luaL_argcheck(L, lua_gettop(L) == 2, 1, "call via '.' and ip version / xdb content buffer expected");
// check the ip version
version = _get_version(L, 1);
if (version == NULL) {
return luaL_error(L, "invalid verison id specified");
}
// content buffer checking
xBuffer = (xdb_buffer_t *) luaL_checkudata(L, 2, XDB_BUFFER_METATABLE_NAME);
if (xBuffer->type != xdb_content_buffer) {
return luaL_error(L, "invalid xdb content buffer");
}
// alloc the searcher
searcher = (xdb_searcher_t *) lua_newuserdata(L, sizeof(xdb_searcher_t));
if (searcher == NULL) {
return luaL_error(L, "failed to alloc xdb searcher entry");
}
// init the xdb searcher
err = xdb_new_with_buffer(version, searcher, (xdb_content_t *) xBuffer->ptr);
if (err != 0) {
lua_pushnil(L);
lua_pushfstring(L, "init content cached xdb searcher with errcode=%d", err);
return 2;
}
// push the metatable onto the stack and
// set it as the metatable of the current searcher
luaL_getmetatable(L, XDB_METATABLE_NAME);
lua_setmetatable(L, -2);
lua_pushnil(L);
return 2;
}
static int lua_xdb_close(lua_State *L) {
xdb_searcher_t *searcher;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via ':'");
searcher = (xdb_searcher_t *) luaL_checkudata(L, 1, XDB_METATABLE_NAME);
if (searcher == NULL) {
return luaL_error(L, "broken xdb searcher instance");
}
xdb_close(searcher);
return 0;
}
static int lua_xdb_search(lua_State *L) {
int err, vid, ip_len;
const char *ip_string;
bytes_ip_t ip_buffer[INET6_ADDRSTRLEN] = {'\0'};
const bytes_ip_t *ip_bytes;
xdb_version_t *version;
xdb_region_buffer_t region;
xdb_searcher_t *searcher;
luaL_argcheck(L, lua_gettop(L) == 2, 2, "call via ':' and string ip address expected");
// get the searcher
searcher = (xdb_searcher_t *) luaL_checkudata(L, 1, XDB_METATABLE_NAME);
ip_string = luaL_checkstring(L, 2);
// ip string type checking
if (strlen(ip_string) < 2) {
lua_pushnil(L);
lua_pushfstring(L, "invalid ip address `%s`", ip_string);
return 2;
}
if (ip_string[0] == '&') {
vid = ip_string[1] & 0xFF;
if (vid == xdb_ipv4_id) {
ip_len = xdb_ipv4_bytes;
} else if (vid == xdb_ipv6_id) {
ip_len = xdb_ipv6_bytes;
} else {
lua_pushnil(L);
lua_pushstring(L, "invalid binary ip bytes specified");
return 2;
}
ip_bytes = ip_string + 2;
// printf("ip_len: %d, vid: %d\n", ip_len, vid);
} else {
version = xdb_parse_ip(ip_string, ip_buffer, sizeof(ip_buffer));
if (version == NULL) {
lua_pushnil(L);
lua_pushfstring(L, "failed to parse string ip `%s`", ip_string);
return 2;
}
ip_len = version->bytes;
ip_bytes = ip_buffer;
}
// init the region buffer
err = xdb_region_buffer_init(&region, NULL, 0);
if (err != 0) {
return luaL_error(L, "failed to init the region buffer with errcode=%d", err);
}
// do the search
err = xdb_search(searcher, ip_bytes, ip_len, &region);
if (err != 0) {
lua_pushinteger(L, err);
lua_pushfstring(L, "err=%d", err);
} else {
lua_pushstring(L, region.value);
lua_pushnil(L);
}
// clean up the region buffer
xdb_region_buffer_free(&region);
return 2;
}
static int lua_xdb_get_io_count(lua_State *L) {
xdb_searcher_t *searcher;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via ':' or xdb searcher was broken");
searcher = (xdb_searcher_t *) luaL_checkudata(L, 1, XDB_METATABLE_NAME);
lua_pushinteger(L, xdb_get_io_count(searcher));
return 1;
}
static int lua_xdb_tostring(lua_State *L) {
xdb_searcher_t *searcher;
luaL_argcheck(L, lua_gettop(L) == 1, 1, "call via ':' or xdb searcher was broken");
searcher = (xdb_searcher_t *) luaL_checkudata(L, 1, XDB_METATABLE_NAME);
lua_pushfstring(L, "xdb %s searcher object", xdb_get_version(searcher)->name);
return 1;
}
// cleanup the current module
static int lua_xdb_cleanup(lua_State *L) {
xdb_clean_winsock();
return 0;
}
// module method define, should be access via ':'
static const struct luaL_Reg xdb_searcher_methods[] = {
{"search", lua_xdb_search},
{"get_io_count",lua_xdb_get_io_count},
{"close", lua_xdb_close},
{"__gc", lua_xdb_close},
{"__tostring", lua_xdb_tostring},
{NULL, NULL},
};
// module function define, should be access via '.'
static const struct luaL_Reg xdb_searcher_functions[] = {
{"new_with_file_only", lua_xdb_new_with_file_only},
{"new_with_vector_index", lua_xdb_new_with_vector_index},
{"new_with_buffer", lua_xdb_new_with_buffer},
{"load_header", lua_xdb_load_header_from_file},
{"load_vector_index", lua_xdb_load_vector_index_from_file},
{"load_content", lua_xdb_load_content_from_file},
{"verify", lua_xdb_verify_from_file},
{"version_from_header", lua_xdb_version_from_header},
{"version_info", lua_xdb_version_info},
{"cleanup", lua_xdb_cleanup},
{"parse_ip", lua_xdb_parse_ip},
{"ip_to_string", lua_xdb_ip_to_string},
{"ip_compare", lua_xdb_ip_compare},
{"now", lua_xdb_now},
{NULL, NULL}
};
// module register function
int luaopen_xdb_searcher(lua_State *L)
{
int err = xdb_init_winsock();
if (err != 0) {
luaL_error(L, "failed to init the winsock with errno=%d\n", err);
return 1;
}
// create a metatable for xdb buffer object
luaL_newmetatable(L, XDB_BUFFER_METATABLE_NAME);
lua_pushvalue(L, -1);
lua_setfield(L, -2, "__index");
luaL_setfuncs(L, xdb_buffer_methods, 0);
// create a metatable for xdb searcher object
luaL_newmetatable(L, XDB_METATABLE_NAME);
lua_pushvalue(L, -1);
lua_setfield(L, -2, "__index");
luaL_setfuncs(L, xdb_searcher_methods, 0);
luaL_setfuncs(L, xdb_searcher_functions, 0);
// register the constants attributes
lua_pushinteger(L, xdb_ipv4_id);
lua_setfield(L, -2, "IPv4");
lua_pushinteger(L, xdb_ipv6_id);
lua_setfield(L, -2, "IPv6");
lua_pushinteger(L, xdb_header_buffer);
lua_setfield(L, -2, "header_buffer");
lua_pushinteger(L, xdb_vector_index_buffer);
lua_setfield(L, -2, "v_index_buffer");
lua_pushinteger(L, xdb_content_buffer);
lua_setfield(L, -2, "content_buffer");
return 1;
}

72
binding/nginx/README.md
View File

@ -1,72 +0,0 @@
# nginx-ip2region
## build
```shell
$ mkdir -p workspace
$ cd workspace
$ wget https://nginx.org/download/nginx-1.23.4.tar.gz
$ tar -zxf nginx-1.23.4.tar.gz && rm -rf nginx-1.23.4.tar.gz
$ git clone https://github.com/lionsoul2014/ip2region.git
$ cd ip2region/binding/c
$ make xdb_searcher_lib
$ cd ../../../nginx-1.23.4
$ ./configure \
--add-module=$(PWD)/../ip2region/binding/nginx \
--with-cc-opt="-I $(PWD)/../ip2region/binding/c/build/include" \
--with-ld-opt="-L$(PWD)/../ip2region/binding/c/build/lib"
$ make
$ make install
```
## nginx conf
> Syntax: `ip2region_db xdb_file_path [cache_policy Optional]`;
> Context: http
cache_policy: `file` or `vectorIndex` or `content`, default: `content`
Edit `nginx.conf` add `ip2region_db` directive
```nginx
...
http {
log_format main escape=json '{'
'"remote_addr": "$remote_addr", '
'"region": "$ip2region", '
'"http_x_forwarded_for": "$http_x_forwarded_for"'
'}';
access_log logs/access.log main;
# set xdb file path
ip2region_db ip2region.xdb;
# ip2region_db ip2region.xdb vectorIndex;
# ip2region_db ip2region.xdb file;
# ip2region_db ip2region.xdb content;
server {
listen 80;
server_name localhost;
location / {
root html;
index index.html index.htm;
}
}
}
```
Copy `ip2region.xdb` to `nginx/config` folder, then restart nginx, the `region` data stored in `ip2region` variable
nginx access log sample
```log
{"remote_addr": "127.0.0.1", "region": "0|0|0|内网IP|内网IP", "http_x_forwarded_for": ""}
{"remote_addr": "127.0.0.1", "region": "0|0|0|内网IP|内网IP", "http_x_forwarded_for": ""}
```
Made with ♥ by Wu Jian Ping

22
binding/nginx/config
View File

@ -1,22 +0,0 @@
ngx_addon_name=ngx_http_ip2region_module
NGX_HTTP_IP2REGION_SRCS=" \
$ngx_addon_dir/src/ngx_http_ip2region_module.c \
"
NGX_HTTP_IP2REGION_DEPS=" \
"
if test -n "$ngx_module_link"; then
ngx_module_type=HTTP
ngx_module_name=$ngx_addon_name
ngx_module_deps="$NGX_HTTP_IP2REGION_DEPS"
ngx_module_srcs="$NGX_HTTP_IP2REGION_SRCS"
ngx_module_libs="-lxdb_searcher"
. auto/module
else
HTTP_MODULES="$HTTP_MODULES $ngx_addon_name"
NGX_ADDON_DEPS="$NGX_ADDON_DEPS $NGX_HTTP_IP2REGION_DEPS"
NGX_ADDON_SRCS="$NGX_ADDON_SRCS $NGX_HTTP_IP2REGION_SRCS"
fi

345
binding/nginx/src/ngx_http_ip2region_module.c
View File

@ -1,345 +0,0 @@
/*
* Created by Wu Jian Ping on - 2023/03/30.
*/
#include "ngx_http_ip2region_module.h"
static ngx_int_t ngx_http_ua_parser_test_full_name(char *name);
static ngx_int_t ngx_http_ip2region_add_variables(ngx_conf_t *cf);
static void *ngx_http_ip2region_create_conf(ngx_conf_t *cf);
static void ngx_http_ip2region_cleanup(void *data);
static char *ngx_http_ip2region_init(ngx_conf_t *cf,
ngx_command_t *cmd, void *conf);
static ngx_int_t ngx_http_ip2region_variable(ngx_http_request_t *r,
ngx_http_variable_value_t *v, uintptr_t data);
static ngx_http_module_t ngx_http_ip2region_ctx = {
ngx_http_ip2region_add_variables, /* pre configuration */
NULL, /* post configuration */
ngx_http_ip2region_create_conf, /* create main configuration */
NULL, /* init main configuration */
NULL, /* create server configuration */
NULL, /* merge server configuration */
NULL, /* create location configuration */
NULL /* merge location configuration */
};
static ngx_command_t ngx_http_ip2region_commands[] = {
{ ngx_string("ip2region_db"),
NGX_HTTP_MAIN_CONF | NGX_CONF_TAKE12,
ngx_http_ip2region_init,
NGX_HTTP_MAIN_CONF_OFFSET,
0,
NULL },
ngx_null_command
};
/* ngx_module_t is required, otherwise failed at complie time */
ngx_module_t ngx_http_ip2region_module = {
NGX_MODULE_V1,
&ngx_http_ip2region_ctx, /* module context */
ngx_http_ip2region_commands, /* module directives */
NGX_HTTP_MODULE, /* module type */
NULL, /* init master */
NULL, /* init module */
NULL, /* init process */
NULL, /* init thread */
NULL, /* exit thread */
NULL, /* exit process */
NULL, /* exit master */
NGX_MODULE_V1_PADDING
};
static ngx_http_variable_t ngx_http_ip2region_vars[] = {
{ ngx_string("ip2region"), NULL,
ngx_http_ip2region_variable,
0, 0, 0 },
ngx_http_null_variable
};
static char *
ngx_http_ip2region_init(ngx_conf_t *cf, ngx_command_t *cmd,
void *conf)
{
ngx_http_ip2region_conf_t *ip2region_cf;
ip2region_searcher_t *ip2region_searcher;
char *db_name, *cache_policy;
ngx_str_t *value;
int err;
char *db_path;
size_t len;
ip2region_cf = conf;
if (ip2region_cf->ip2region_searcher) {
return "is duplicate";
}
value = cf->args->elts;
db_name = (char *)value[1].data;
// default cache_policy: content
if(cf->args->nelts == 2) {
cache_policy = "content";
} else {
cache_policy = (char *)value[2].data;
}
if(ngx_http_ua_parser_test_full_name(db_name) == NGX_OK) {
db_path = db_name;
} else { // relative path to conf directory
len = ngx_cycle->conf_prefix.len + strlen(db_name) + 1;
db_path = malloc(len);
memset(db_path, '\0', len);
memcpy(db_path, ngx_cycle->conf_prefix.data, ngx_cycle->conf_prefix.len);
strcat(db_path, db_name);
}
ip2region_searcher = ngx_palloc(cf->pool, sizeof(ip2region_searcher_t));
if(ip2region_searcher == NULL) {
return NGX_CONF_ERROR;
}
ip2region_searcher->v_index = NULL;
ip2region_searcher->c_buffer = NULL;
if (strcmp(cache_policy, "file") == 0) {
err = xdb_new_with_file_only(&ip2region_searcher->searcher, db_path);
if (err != 0) {
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"failed to create searcher: %s", db_path);
return NGX_CONF_ERROR;
}
} else if (strcmp(cache_policy, "vectorIndex") == 0) {
ip2region_searcher->v_index = xdb_load_vector_index_from_file(db_path);
if (ip2region_searcher->v_index == NULL) {
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"failed to load vector index from: %s", db_path);
return NGX_CONF_ERROR;
}
err = xdb_new_with_vector_index(&ip2region_searcher->searcher, db_path, ip2region_searcher->v_index);
if (err != 0) {
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"failed to create vector index cached searcher: %s", db_path);
return NGX_CONF_ERROR;
}
} else if (strcmp(cache_policy, "content") == 0) {
ip2region_searcher->c_buffer = xdb_load_content_from_file(db_path);
if (ip2region_searcher->c_buffer == NULL) {
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"failed to load xdb content: %s", db_path);
return NGX_CONF_ERROR;
}
err = xdb_new_with_buffer(&ip2region_searcher->searcher, ip2region_searcher->c_buffer);
if (err != 0) {
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"failed to create content cached searcher: %s", db_path);
return NGX_CONF_ERROR;
}
} else {
ngx_conf_log_error(NGX_LOG_EMERG, cf, 0,
"invalid cache policy `%V`, options: file/vectorIndex/content", &value[2]);
return NGX_CONF_ERROR;
}
ip2region_cf->ip2region_searcher = ip2region_searcher;
if(ngx_http_ua_parser_test_full_name(db_name) != NGX_OK) {
free(db_path);
}
return NGX_CONF_OK;
}
static void *
ngx_http_ip2region_create_conf(ngx_conf_t *cf)
{
ngx_pool_cleanup_t *cln;
ngx_http_ip2region_conf_t *conf;
conf = ngx_pcalloc(cf->pool, sizeof(ngx_http_ip2region_conf_t));
if (conf == NULL) {
return NULL;
}
cln = ngx_pool_cleanup_add(cf->pool, 0);
if (cln == NULL) {
return NULL;
}
cln->handler = ngx_http_ip2region_cleanup;
cln->data = conf;
return conf;
}
static ngx_int_t
ngx_http_ip2region_add_variables(ngx_conf_t *cf)
{
ngx_http_variable_t *var;
ngx_http_variable_t *v;
for (v = ngx_http_ip2region_vars; v->name.len; v++) {
var = ngx_http_add_variable(cf, &v->name, v->flags);
if (var == NULL) {
return NGX_ERROR;
}
var->get_handler = v->get_handler;
var->data = v->data;
}
return NGX_OK;
}
static ngx_int_t
ngx_http_ip2region_variable(ngx_http_request_t *r,
ngx_http_variable_value_t *v, uintptr_t data)
{
ngx_http_ip2region_conf_t *ip2region_conf;
struct sockaddr_in *sin;
char region[512] = {'\0'};
int err;
unsigned int ip;
#if (NGX_HAVE_INET6)
u_char *p;
in_addr_t addr;
struct sockaddr_in6 *sin6;
#endif
ip2region_conf = ngx_http_get_module_main_conf(r, ngx_http_ip2region_module);
if (ip2region_conf->ip2region_searcher != NULL) {
switch (r->connection->sockaddr->sa_family) {
case AF_INET:
sin = (struct sockaddr_in *) r->connection->sockaddr;
ip = htonl(sin->sin_addr.s_addr);
err = xdb_search(&ip2region_conf->ip2region_searcher->searcher, ip, region, sizeof(region));
if (err == 0) {
v->data = (unsigned char *)region;
v->len = strlen(region);
return NGX_OK;
}
break;
#if (NGX_HAVE_INET6)
case AF_INET6:
sin6 = (struct sockaddr_in6 *) r->connection->sockaddr;
p = sin6->sin6_addr.s6_addr;
if (IN6_IS_ADDR_V4MAPPED(&sin6->sin6_addr)) {
addr = p[12] << 24;
addr += p[13] << 16;
addr += p[14] << 8;
addr += p[15];
ip = htonl(addr);
err = xdb_search(&ip2region_conf->ip2region_searcher->searcher, ip, region, sizeof(region));
if (err == 0) {
v->data = (unsigned char *)region;
v->len = strlen(region);
return NGX_OK;
}
}
break;
#endif
}
}
v->not_found = 1;
return NGX_OK;
}
static void
ngx_http_ip2region_cleanup(void *data)
{
ngx_http_ip2region_conf_t *ip2region_conf = data;
if(ip2region_conf->ip2region_searcher != NULL) {
xdb_close(&ip2region_conf->ip2region_searcher->searcher);
// check and free the vector index
if (ip2region_conf->ip2region_searcher->v_index != NULL) {
xdb_close_vector_index(ip2region_conf->ip2region_searcher->v_index);
ip2region_conf->ip2region_searcher->v_index = NULL;
}
// check and free the content buffer
if (ip2region_conf->ip2region_searcher->c_buffer != NULL) {
xdb_close_content(ip2region_conf->ip2region_searcher->c_buffer);
ip2region_conf->ip2region_searcher->c_buffer = NULL;
}
ip2region_conf->ip2region_searcher = NULL;
}
}
static ngx_int_t
ngx_http_ua_parser_test_full_name(char *name)
{
#if (NGX_WIN32)
u_char c0, c1;
c0 = name[0];
if (strlen(name) < 2) {
if (c0 == '/') {
return 2;
}
return NGX_DECLINED;
}
c1 = name[1];
if (c1 == ':') {
c0 |= 0x20;
if ((c0 >= 'a' && c0 <= 'z')) {
return NGX_OK;
}
return NGX_DECLINED;
}
if (c1 == '/') {
return NGX_OK;
}
if (c0 == '/') {
return 2;
}
return NGX_DECLINED;
#else
if (name[0] == '/') {
return NGX_OK;
}
return NGX_DECLINED;
#endif
}

23
binding/nginx/src/ngx_http_ip2region_module.h
View File

@ -1,23 +0,0 @@
/*
* Created by Wu Jian Ping on - 2023/03/30.
*/
#ifndef __NGX_HTTP_IP2REGION_MODULE_H_INCLUDED__
#define __NGX_HTTP_IP2REGION_MODULE_H_INCLUDED__
#include <ngx_config.h>
#include <ngx_core.h>
#include <ngx_http.h>
#include <xdb_searcher.h>
typedef struct {
xdb_searcher_t searcher;
xdb_vector_index_t *v_index;
xdb_content_t *c_buffer;
} ip2region_searcher_t;
typedef struct {
ip2region_searcher_t *ip2region_searcher;
} ngx_http_ip2region_conf_t;
#endif

27
binding/nginx/t/http_ip2region.t
View File

@ -1,27 +0,0 @@
use lib 'lib';
use Test::Nginx::Socket; # 'no_plan';
repeat_each(2);
plan tests => repeat_each() * 124;
no_long_string();
#no_diff;
run_tests();
__DATA__
=== TEST 1: set request header at client side
--- config
location /foo {
echo $http_x_foo;
}
--- request
GET /foo
--- more_headers
X-Foo: blah
--- response_headers
! X-Foo
--- response_body
blah

17
binding/nodejs/.eslintrc.js
View File

@ -1,17 +0,0 @@
module.exports = {
env: {
browser: true,
commonjs: true,
es2021: true
},
parserOptions: {
ecmaVersion: 'latest'
},
extends: [
'standard'
],
globals: {
describe: true,
it: true
}
}

5
binding/nodejs/.mocharc.json
View File

@ -1,5 +0,0 @@
{
"reporter": "spec",
"timeout": 60000,
"exit": true
}

5
binding/nodejs/.nycrc.json
View File

@ -1,5 +0,0 @@
{
"all": true,
"report-dir": "./coverage/",
"reporter": ["text", "html"]
}

155
binding/nodejs/ReadMe.md
View File

@ -1,155 +0,0 @@
# ip2region nodejs 查询客户端实现
## 使用方式
### 完全基于文件的查询
```javascript
// 导入包
const Searcher = require('.')
// 指定ip2region数据文件路径
const dbPath = 'ip2region.xdb file path'
try {
// 创建searcher对象
const searcher = Searcher.newWithFileOnly(dbPath)
// 查询
const data = await searcher.search('218.4.167.70')
// data: {region: '中国|0|江苏省|苏州市|电信', ioCount: 3, took: 1.342389}
} catch(e) {
console.log(e)
}
```
### 缓存 `VectorIndex` 索引
```javascript
// 导入包
const Searcher = require('.')
// 指定ip2region数据文件路径
const dbPath = 'ip2region.xdb file path'
try {
// 同步读取vectorIndex
const vectorIndex = Searcher.loadVectorIndexFromFile(dbPath)
// 创建searcher对象
const searcher = Searcher.newWithVectorIndex(dbPath, vectorIndex)
// 查询 await 或 promise均可
const data = await searcher.search('218.4.167.70')
// data: {region: '中国|0|江苏省|苏州市|电信', ioCount: 2, took: 0.402874}
} catch(e) {
console.log(e)
}
```
### 缓存整个 `xdb` 数据
```javascript
// 导入包
const Searcher = require('.')
// 指定ip2region数据文件路径
const dbPath = 'ip2region.xdb file path'
try {
// 同步读取buffer
const buffer = Searcher.loadContentFromFile(dbPath)
// 创建searcher对象
const searcher = Searcher.newWithBuffer(buffer)
// 查询 await 或 promise均可
const data = await searcher.search('218.4.167.70')
// data: {region:'中国|0|江苏省|苏州市|电信', ioCount: 0, took: 0.063833}
} catch(e) {
console.log(e)
}
```
## 查询测试
可以通过 `node ./tests/test.app.js` 命令来测试查询：
```shell
➜ nodejs git:(v2.0-for-nodejs) ✗ node ./tests/test.app.js --help
usage: Usage node test.app.js <agrs>
ip2region test app
optional arguments:
-h, --help show this help message and exit
-d DB, --db DB ip2region binary xdb file path, default: ../../data/ip2region.xdb
-c CACHE_POLICY, --cache-policy CACHE_POLICY
cache policy: file/vectorIndex/content, default: content
```
例如：使用默认的 data/ip2region.xdb 文件进行查询测试：
```shell
➜ nodejs git:(v2.0-for-nodejs) ✗ node ./tests/test.app.js
parameters:
dbPath: ../../data/ip2region.xdb
cache-policy: content
type 'quit' to exit
ip2region>> 1.2.3.4
{ region: '美国|0|华盛顿|0|谷歌', ioCount: 0, took: 54.606261 }
ip2region>>
```
输入 ip 即可进行查询测试，也可以分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的查询效果。
## bench 测试
```shell
➜ nodejs git:(v2.0-for-nodejs) ✗ node ./tests/bench.app.js --help
usage: Usage node test.app.js [command options]
ip2region benchmark app
optional arguments:
-h, --help show this help message and exit
--db DB ip2region binary xdb file path, default: ../../data/ip2region.xdb
--src SRC source ip text file path, default: ../../data/ip.merge.txt
--cache-policy CACHE_POLICY
cache policy: file/vectorIndex/content, default: content
```
例如：通过默认的 data/ip2region.xdb 和 data/ip.merge.txt 文件进行 bench 测试：
```shell
➜ nodejs git:(v2.0-for-nodejs) ✗ node ./tests/bench.app.js
options:
dbPath: ../../data/ip2region.xdb
src: ../../data/ip2region.xdb
cache-policy: content
Bench finished, {cachePolicy: content, total: 3417955, took: 20.591887765s, cost: 6.02462225658325μs/op}
```
可以通过分别设置 `cache-policy` 为 file/vectorIndex/content 来测试三种不同缓存实现的效果。
>Note: 注意 bench 使用的 src 文件要是生成对应 xdb 文件相同的源文件。
## 单元测试及覆盖率结果
```shell
➜ nodejs git:(v2.0-for-nodejs) ✗ npm run coverage
...
ip2region
✔ #newWithFileOnly and search
✔ #newWithVectorIndex and search
✔ #newWithBuffer and search
3 passing (6ms)
----------|---------|----------|---------|---------|----------------------------------
File | % Stmts | % Branch | % Funcs | % Lines | Uncovered Line #s
----------|---------|----------|---------|---------|----------------------------------
All files | 91.58 | 60.71 | 100 | 91.58 |
index.js | 91.58 | 60.71 | 100 | 91.58 | 61,75,90,146,152,187,193,207,215
----------|---------|----------|---------|---------|----------------------------------
```
Made with ♥ by Wu Jian Ping

Some files were not shown because too many files have changed in this diff Show More