Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[文档] ldap 配置/使用/FAQ 汇总 #429

Open
wklken opened this issue May 12, 2022 · 45 comments
Open

[文档] ldap 配置/使用/FAQ 汇总 #429

wklken opened this issue May 12, 2022 · 45 comments
Assignees
Labels
Module: LDAP/MAD ldap/mad related Module: plugin DataSource Plugin Sign: help wanted Extra attention is needed Type: documentation Improvements or additions to documentation Type: question Usage or problems

Comments

@wklken
Copy link
Collaborator

wklken commented May 12, 2022

背景: ldap由于是外部插件的形式, 内部根本没有场景, 而且ldap server多种多样, 配置差异很大, 非常不好配置; Github issues和社区问答中都存在很多的问题咨询和解答.

目标: 汇总所有相关问题, 并实时更新, 便于用户检索和解决问题


相关入口

  1. 如何对接已有的 LDAP 用户体系 了解概念和流程
  2. 如何对接已有的 MAD 用户体系 了解概念和流程
  3. 蓝鲸问答社区: 用户管理LDAP,AD同步FAQ 如何配置/具体配置示例
  4. 蓝鲸问答社区: 【长期更新】社区版6.0~6.0.3 用户管理问题解答汇总 / 蓝鲸问答社区: 【长期更新】社区版6.0.4 用户管理问题解答汇总 / 【每日更新】社区版6.0常见问题汇总
  5. 蓝鲸问答社区:社区版常见问题汇总 / 经验分享/Bug 处理/解决方案

如何使用

  1. 如果没有相关的背景知识(例如ldap是什么,怎么配置), 请先阅读相关入口部分的相关文档1/2/3
  2. ctrl+f在本页搜索相关的关键字, 欢迎补充相关答案
  3. 如果本页没有, 可以到相关入口的`4/5进一步查找(因为更新同步存在延时)
  4. 如果问答社区也没有, 请在社区问答开一个新帖, 会有专门的同学负责跟进, 协助解决

欢迎补充相关的 FAQ 及解答, 分享具体实践的经验

@wklken wklken added Type: documentation Improvements or additions to documentation Layer: api Api module related Type: proposal Proposal Module: LDAP/MAD ldap/mad related labels May 12, 2022
@wklken wklken self-assigned this May 12, 2022
@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

ldap地址配置正确, 但是就是无法测试连接一直失败(用户管理版本<=2.3.3)

ldap服务端: OpenLDAP, 启动时mode=ipv4_only (-4 Listen on IPv4 addresses only)

报错

Traceback (most recent call last):
  File "/app/bkuser_core/categories/plugins/ldap/client.py", line 71, in initialize
    return Connection(**connection_params)
  File "/usr/local/lib/python3.6/site-packages/ldap3/core/connection.py", line 326, in __init__
    self.do_auto_bind()
  File "/usr/local/lib/python3.6/site-packages/ldap3/core/connection.py", line 341, in do_auto_bind
    self.open(read_server_info=False)
  File "/usr/local/lib/python3.6/site-packages/ldap3/strategy/sync.py", line 56, in open
    BaseStrategy.open(self, reset_usage, read_server_info)
  File "/usr/local/lib/python3.6/site-packages/ldap3/strategy/base.py", line 150, in open
    raise LDAPSocketOpenError('invalid server address')
ldap3.core.exceptions.LDAPSocketOpenError: invalid server address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/bkuser_core/categories/views.py", line 197, in test_connection
    syncer_cls(instance.id).fetcher.client.initialize(**serializer.validated_data)
  File "<string>", line 6, in __init__
  File "/app/bkuser_core/categories/plugins/ldap/syncer.py", line 159, in __post_init__
    self.fetcher: LDAPFetcher = self.get_fetcher()
  File "/app/bkuser_core/categories/plugins/base.py", line 230, in get_fetcher
    return self.fetcher_cls(self.category_id, self.config_loader)
  File "<string>", line 4, in __init__
  File "/app/bkuser_core/categories/plugins/ldap/syncer.py", line 40, in __post_init__
    self.client = LDAPClient(self.config_loader)
  File "<string>", line 3, in __init__
  File "/app/bkuser_core/categories/plugins/ldap/client.py", line 38, in __post_init__
    use_ssl=bool(self.config_provider.get("ssl_encryption") == "SSL"),
  File "/app/bkuser_core/categories/plugins/ldap/client.py", line 79, in initialize
    raise local_exceptions.LdapCannotBeInitialized
bkuser_core.categories.plugins.ldap.exceptions.LdapCannotBeInitialized

处理:

进入用户管理容器或虚拟环境, 升级ldap32.7 解决; 点击连接测试, 测试通过

pip install ldap3==2.7

相关issue:

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

有没有python脚本可以快速测试配置的 ldap 服务是否可用?

TODO

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

ldap3.core.exceptions.LDAPBindError

使用openldap账户@登录域 的用户名和密码登不上环境; 登录提示: 账户名和密码不匹配

报错日志:

INFO [2022-04-11 12:08:04] bkuser.categories.vendors.ldap.client(ln:74): going to search (objectClass=inetOrgPerson) from ou=rd_xxxn,ou=rd,ou=People,dc=test,dc=com
ERROR [2022-04-11 12:08:04] bkuser.profiles.views(ln:529): check profile<bkdevops> failed
Traceback (most recent call last):
  File "/data/bkce/usermgr/api/bkuser/profiles/views.py", line 527, in login
  File "/data/bkce/usermgr/api/bkuser/categories/vendors/ldap/login.py", line 38, in check
  File "/data/bkce/usermgr/api/bkuser/categories/vendors/ldap/client.py", line 94, in check
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/ldap3/core/connection.py", line 326, in __init__
    self.do_auto_bind()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/ldap3/core/connection.py", line 354, in do_auto_bind
    raise LDAPBindError(self.last_error)
ldap3.core.exceptions.LDAPBindError: automatic bind not successful - invalidCredentials

image

页面报错提示:

LDAP服务器连接失败 (ldap3.core.exceptions.LDAPBindError: automatic bind not successful - invalidCredentials)

原因: 配置的 LDAP 管理密码有误

相关issue

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

OpenLDAP导入的账户,页面"全名"字段为空

image

解决方案:在LDAP字段配置 -> 中文名处设置

image

相关issue:

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

ldap用的ssl, 如何配置同步

ldap 地址规则:

  • 不使用 ssl 时,应该写成 ldap://x.x.x.x:389 (注意这里端口需要根据自己环境服务提供端口修改)
  • 使用 ssl 时,需要写成 ldaps://x.x.x.x:636 (注意这里端口需要根据自己环境服务提供端口修改)
    • 注意, 此时页面配置时, SSL 加密方式需要下拉选择SSL

image

配置错误可能的报错:

image

相关issue:

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

ldap配置同步用户及部门成功了, 但是登录不上 (LDAP用户登录提示密码错误)

新建 ldap 同步目录的时候, 会有一个(domain), 例如abc, domain是目录创建时指定的唯一标识

登录目前的交互是需要, 用户名输入框需要用户主动填写, 否则登录校验走默认目录

格式: {identity}@域, 邮箱除外(邮箱相当于自带了域, 即, 想支持邮箱登录, 邮箱后缀需要与ldap目录的一致)

所以登录时, ldap 同步过来的用户, 用户名需要填写用户名@abc/手机号@abc/邮箱

image

相关issue:

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

ldap插件同步的代码在哪里? 使用了什么库?

https://github.com/TencentBlueKing/bk-user/tree/development/src/api/bkuser_core/categories/plugins/ldap

其中, 测试连接的逻辑代码是

try:
if use_ssl:
server_params.update({"use_ssl": True})
connection_params = {
"server": Server(connection_url, **server_params),
"auto_bind": True,
"receive_timeout": timeout_setting,
}
if hasattr(settings, "LDAP_CONNECTION_EXTRAS_PARAMS"):
connection_params.update(settings.LDAP_CONNECTION_EXTRAS_PARAMS)
# 目前只支持简单鉴权
if user and password:
connection_params.update({"user": user, "password": password, "authentication": SIMPLE})
return Connection(**connection_params)
except KeyError:
logger.exception("failed to initialize ldap server. KeyError. [url=%s]", connection_url)
raise local_exceptions.LDAPSettingNotReady
except ldap3.core.exceptions.LDAPSocketReceiveError:
logger.exception("failed to initialize ldap server. LDAPSocketReceiveError. [url=%s]", connection_url)
raise local_exceptions.LdapCannotBeInitialized
except Exception:
logger.exception("failed to initialize ldap server. [url=%s]", connection_url)
raise local_exceptions.LdapCannotBeInitialized

目前依赖的是第三方库 ldap3

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

用户管理是否支持过滤 Microsoft Active Directory 禁用用户

#178

  1. 用户管理-AD目录同步的用户对象配置是支持过滤的,过滤使用ad原生语法;
  2. 要进行过滤禁用用户后同步,可以参考下面的配置,在用户目录-字段配置-用户字段类 增加(!(userAccountControl=514))
    image

另:不是所有的AD系统 禁用用户的属性值(userAccountControl)都是514,存在某些AD系统使用60082代表禁用值,60080代表启用值;因此,具体实施还要根据使用的AD系统来进行配置;

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

TODO

#【MAD登录】 MAD同步成功,但域账号登录时仍然提示“账户名和密码不匹配 262

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

修改密码无法同步到 AD 域

Q: 用户集成了Windows AD,但是修改的密码无法同步到AD域里,请问可以实现密码同步更能嘛?
A: 我们目前不会向 AD 做任何的写操作,建议直接在 AD 中修改密码

相关issue: #313

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

对接单点登录之后如何新增admin超级管理员?

Q: 对接公司内部的单点系统之后,所有用户登录都是普通用户,admin不能登录了
A: 需要提前进入用户管理后台 Python 环境(具体步骤依赖部署方式)

from bkuser_core.profiles.models import Profile
from bkuser_core.categories.models import ProfileCategory

p = ProfileCategory.objects.get(id=${期望目录id})
Profile.objects.create(username="admin", domain=p.domain, category_id=p.id)

相关issue: #314

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

LDAP配置的地址是什么格式?

ldap 地址规则:

  • 不使用 ssl 时,应该写成 ldap://x.x.x.x:389 (注意这里端口需要根据自己环境服务提供端口修改)
  • 使用 ssl 时,需要写成 ldaps://x.x.x.x:636 (注意这里端口需要根据自己环境服务提供端口修改)
    • 注意, 此时页面配置时, SSL 加密方式需要下拉选择SSL

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

ldap3.core.exceptions.LDAPSocketOpenError 应该如何测试配置地址是否正确

需要在用户管理(后台api)部署的机器/容器上验证:

  • 如果是二进制版, 需要到用户管理后台部署机器;
  • 如果是容器化版, 需要登入容器内;
  1. 确定配置的ldap 地址及端口号 (以下仅为示例, 需要根据具体配置修改命令后执行)
  2. 使用nc确定
    • 2.1 如果是LDAP 389, 命令: nc <ldapserverip> 389 -v -w 60
    • 2.2 如果是LDAPs 636, 命令: nc <ldapserverip> 636 -v -w 60
  3. 使用telnet确定
    • 3.1 telnet <ldap-server-fqdn> <ldap-port>
  4. 使用curl确定: curl -v IP:389

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

需要补充:

  • 字段基本含义/配置值基本含义
  • 配置基本要求, 例如用户名唯一
  • 不同ldap服务端配置的一些差别和例外(例如memberof)

https://bk.tencent.com/s-mart/community/question/1901

配置说明: 用户基础字段

image

配置说明: 用户扩展字段

image

配置说明: 用户组字段

image

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

用户数据同步成功, 但是登录报错KeyError: 'entrydn'

报错详情

Traceback (most recent call last):
  File "/data/bkce/usermgr/api/bkuser_core/profiles/views.py", line 620, in login
    login_class().check(profile, password)
  File "/data/bkce/usermgr/api/bkuser_core/categories/plugins/ldap/login.py", line 45, in check
    target_dn = self.fetch_dn(user)
  File "/data/bkce/usermgr/api/bkuser_core/categories/plugins/ldap/login.py", line 26, in fetch_dn
    return force_str(user_info["raw_attributes"]["entryDN"][0])
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/ldap3/utils/ciDict.py", line 68, in __getitem__
    return self._store[self._case_insensitive_keymap[self._ci_key(key)]]
KeyError: 'entrydn'
ERROR [2021-10-28 14:44:25] bkuser_core.common.exception_handler(ln:81): request apiServer failed
Traceback (most recent call last):
  File "/data/bkce/usermgr/api/bkuser_core/profiles/views.py", line 620, in login
    login_class().check(profile, password)
  File "/data/bkce/usermgr/api/bkuser_core/categories/plugins/ldap/login.py", line 45, in check
    target_dn = self.fetch_dn(user)
  File "/data/bkce/usermgr/api/bkuser_core/categories/plugins/ldap/login.py", line 26, in fetch_dn
    return force_str(user_info["raw_attributes"]["entryDN"][0])
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/ldap3/utils/ciDict.py", line 68, in __getitem__
    return self._store[self._case_insensitive_keymap[self._ci_key(key)]]
KeyError: 'entrydn'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/data/bkce/usermgr/api/bkuser_core/profiles/views.py", line 635, in login
    raise error_codes.PASSWORD_ERROR
bkuser_core.common.error_codes.CoreAPIError: CoreAPIError 400-PASSWORD_ERROR```

原因: 从 LDAP 服务器中读取出的用户属性中没有 entryDN;

处理: 需要确认一下你们使用的是 OpenLDAP 还是 Microsoft Active Directory, 二者的配置会有些差异
最终确认, 服务器是 AD, 但在用户管理配置成了 LDAP 导致的; 使用 AD 接入, 调试成功

相关issue:

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

支持同步没有开启memberOf模块的openldap

问题: 用户的openldap服务没有开启memberOf模块,用户管理的ldap同步会失败,提示没有memberOf属性。
并非所有用户需要开启memberOf模块,即使后续开启memberOf模块,对openldap原先的存量用户也不起作用,造成一些没有memberOf的的openldap用户无法使用用户管理ldap同步功能

报错详情:

Traceback (most recent call last):
 ...
  raise LDAPAttributeError('invalid attribute type ' + attribute_name_to_check)
ldap3.core.exceptions.LDAPAttributeError: invalid attribute type memberOf

先确认用户管理ldap配置是否正确

image

如果升级到了最新版,这里用户组关联字段是可以不填写的,留空即不同步用户组数据


解决问题的思路:

  1. 如果方便开启 memberOf ,并且可以刷新数据的情况,可以参考 https://blog.adimian.com/2014/10/15/how-to-enable-memberof-using-openldap/ 操作 ldap 服务器
  2. LDAP/MAD 插件支持不填写用户组配置,表示并不需要同步相关数据,用户与组织信息只需要 OU 即可

另外, 如果使用的 LDAP 是由类似 OpenDJ 这样的服务提供

企业微信截图_16354860654024

那么你需要手动修改用户组默认属性配置,由 memberOf 修改为 isMemberOf

进入 Api 模块的 Python Shell 环境:

from bkuser_core.user_settings.models import Setting
# 填写相关联的目录 ID
s = Setting.objects.get(category_id=${category_id}, meta__key="user_member_of")
s.value = "isMemberOf"
s.save()

相关issue:

@wklken
Copy link
Collaborator Author

wklken commented May 13, 2022

ldap 同步部门成功, 但是同步用户失败username<小明> does not meet format

报错详情:

[2022-05-12 16:20:24,212: INFO/ForkPoolWorker-9] 同步总耗时: 2.444955825805664s, 消耗总CPU时间: 0.175728s.
[2022-05-12 16:29:04,212: WARNING/ForkPoolWorker-11] username<小明> does not meet format

username does not meet format
  1. 原因: 用户名配置的映射是中文名(displayName)

解决: 改成uid后同步成功

image

  1. 原因: 用户名不符合格式要求

username 命名规范: 由1-32位字母、数字、下划线(_)、点(.)、减号(-)字符组成,以字母或数字开头

不能包含空格/中文/特殊字符

@wklken
Copy link
Collaborator Author

wklken commented May 17, 2022

FAQ: 配置mad, 同步用户出错

报错详情:

bkuser.categories.exceptions.FetchDataFromRemoteFailed: 无法获取用户数据, 请检查配置

@wklken
Copy link
Collaborator Author

wklken commented May 17, 2022

版本bug: ldap目录下, 用户绑定的企业微信, ldap同步之后被清空 version < 2.2.6-b3

@wklken
Copy link
Collaborator Author

wklken commented May 17, 2022

FAQ: 同一蓝鲸平台可以同时接入多个域控吗>

可以, 使用多个目录, 每个目录配置不同域, 使用ldap分别同步即可

登录时, 用户名为 username@域

相关issue: https://bk.tencent.com/s-mart/community/question/2049?type=answer

@wklken
Copy link
Collaborator Author

wklken commented May 17, 2022

MAD/LDAP同步报错: Duplicate entry 'xxxxx' for key 'code'

报错详情:

pymysql.err.IntegrityError: (1062, "Duplicate entry 'xxxxx' for key 'code'")

报错原因 1: ldap 中 存在某个用户的 dn 冲突了

相关issue:

报错原因 2: 部门的code冲突

报错日志中包含: Department bulk_create failed 关键字

由于 2.5 及之前的版本, 目录的删除是软删除, 导致用户可能新建一个测试目录, 配置ldap测试同步, 之后删除这个目录, 新建一个正式的目录, 配置同一个ldap源

注意, 软删除之后目前在产品页面是看不到的, 无法恢复或者硬删除, #441 (后续版本会解决)

并且, 2.5 及之前的版本, 存在一个bug #714 (后续版本会解决), 使用的部门full_name作为的code数据库唯一约束(例如: 总公司/研发部),

这样就会导致出现这个冲突的原因有很多:

  1. 同一个目录存在相同的两个部门(两个 总公司/研发部, 概率比较低)
  2. 两个目录, 配置同一个ldap(大概率), 例如上面提到的先配置一个测试目录后删除的场景
  3. 两个目录, 配置不同ldap, 两个ldap存在相同的部门full_name

注意, 如果报 1406, Data too long for column 'code', 可以尝试把部门表的code先改长再同步试试

alter table departments_department modify `code` varchar(256) DEFAULT NULL

解决方案

  1. 如果是确认 原因 1 导致的, 需要从ldap服务数据本身解决, 源头就应该不能有同一个节点下有两个同名的部门
  2. 如果确认是 原因 2 导致的
    • 2.1 如果是新装环境或测试环境, 还没开始使用, 可以截图已调通的ldap配置; 然后重置掉用户管理数据库, 重新安装用户管理(注意如果组织架构已经同步到权限中心, 权限中心也需要重置), 然后重新配置ldap同步;
    • 2.2 从数据库层面进行硬删除 不需要的目录
  3. 如果确认是 原因 3导致的, 看看是否能避免从两个ldap同步数据到同一个用户管理; 如果确实有这种需求, 需要等待用户管理后续版本修正 [bug] ldap用户同步的code目前使用的是full_name #714

@wklken
Copy link
Collaborator Author

wklken commented May 17, 2022

FAQ: 社区版 6.0.2 中用户管理 (2.2.2) 接入 AD/LDAP 用户体系时同步用户异常 (version==2.2.2)

修复: 使用version >=2.2.4进行修复

相关issue:

@wklken
Copy link
Collaborator Author

wklken commented Jun 27, 2022

排查同步入库报错问题: 目录/用户/部门相关表的查询sql

注意, 仅用于排查问题时的查询, 不要修改任何未经过确认的字段值, 不要删除任何数据;(以上都可能导致同步失败等各种奇奇怪怪的问题, 并且可能造成数据丢失无法恢复)

报错关键字:

  • Duplicate entry '' for key 'code'

报错原因:

  • 手工插入用户/部门数据
  • 手工修改了用户/部门数据
  • [大概率] 多个目录配置相同的 ldap, 同步发现冲突
  • [大概率] 新建一个目录配置了ldap, 同步后, 删除该目录, 然后新建另一个目录配置了同一个ldap
  • [大概率] 同一个目录, 配置了 ldap 同步后, 又改了配置, 再次同步
  • ldap服务端存在重复的用户

注意: 只要code不变, 那么在权限中心等, 是同一个用户;

code的生成规则:

  • sha256(category_id + ldap_object[dn])

所以, 如果服务端dn重复, 会导致同步失败; 如果往目录 1 配置同步过一次, 后来改了配置又同步, 存在重复dn也会冲突;


查看目录列表 (注意: status=normal, enabled=1才会在页面上展示)

select * from categories_profilecategory;

查看目录的配置

select b.key, a.value, a.enabled, a.category_id, a.meta_id from user_settings_setting a, user_settings_settingmeta b where a.meta_id=b.id
and a.category_id=123;

查看某个用户 (注意查看status/staff_status/enabled几个字段)

# code / username / display_name
select * from profiles_profile where code='xxxxxx';
select * from profiles_profile where username='xxxxx'; 

select * from profiles_profile where category_id=2 limit 10;

查看某个部门的信息

select * from departments_department where name='DEPT_NAME ';

select * from departments_department where category_id=2 limit 10;

查询部门-用户关系表

select * from departments_department_profiles where category_id=2;

# 部门下的所有用户
select * from departments_department_profiles where department_id=123;

# 用户的所有部门
select * from departments_department_profiles where profile_id=123;

处理:

  1. 如果非生产环境, 并且刚部署没有配置一系列权限数据, 可以进行

@wklken
Copy link
Collaborator Author

wklken commented Jun 27, 2022

部门及用户同步后的code生成规则

def _get_code(self, raw_obj: dict) -> str:
"""通过对象 dn 生成 唯一code"""
dn = f"{self.category_id}-{raw_obj.get('dn')}"
sha = hashlib.sha256(force_bytes(dn)).hexdigest()
logger.info("use dn to be code: %s -> %s", dn, sha)
return sha

@wklken
Copy link
Collaborator Author

wklken commented Jun 27, 2022

ldap地址正确但是测试连接失败 LdapCannotBeInitialized

#428

Traceback (most recent call last):
  File "/app/bkuser_core/categories/plugins/ldap/client.py", line 71, in initialize
    return Connection(**connection_params)
  File "/usr/local/lib/python3.6/site-packages/ldap3/core/connection.py", line 326, in __init__
    self.do_auto_bind()
  File "/usr/local/lib/python3.6/site-packages/ldap3/core/connection.py", line 341, in do_auto_bind
    self.open(read_server_info=False)
  File "/usr/local/lib/python3.6/site-packages/ldap3/strategy/sync.py", line 56, in open
    BaseStrategy.open(self, reset_usage, read_server_info)
  File "/usr/local/lib/python3.6/site-packages/ldap3/strategy/base.py", line 150, in open
    raise LDAPSocketOpenError('invalid server address')
ldap3.core.exceptions.LDAPSocketOpenError: invalid server address

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/app/bkuser_core/categories/views.py", line 197, in test_connection
    syncer_cls(instance.id).fetcher.client.initialize(**serializer.validated_data)
  File "<string>", line 6, in __init__
  File "/app/bkuser_core/categories/plugins/ldap/syncer.py", line 159, in __post_init__
    self.fetcher: LDAPFetcher = self.get_fetcher()
  File "/app/bkuser_core/categories/plugins/base.py", line 230, in get_fetcher
    return self.fetcher_cls(self.category_id, self.config_loader)
  File "<string>", line 4, in __init__
  File "/app/bkuser_core/categories/plugins/ldap/syncer.py", line 40, in __post_init__
    self.client = LDAPClient(self.config_loader)
  File "<string>", line 3, in __init__
  File "/app/bkuser_core/categories/plugins/ldap/client.py", line 38, in __post_init__
    use_ssl=bool(self.config_provider.get("ssl_encryption") == "SSL"),
  File "/app/bkuser_core/categories/plugins/ldap/client.py", line 79, in initialize
    raise local_exceptions.LdapCannotBeInitialized
bkuser_core.categories.plugins.ldap.exceptions.LdapCannotBeInitialized

ldap3从 2.6.1, 升级到 2.7 之后解决;

@wklken
Copy link
Collaborator Author

wklken commented Jun 29, 2022

illegal mix of collations for operation 'case'

问题: 同步时, 入数据库pymysql报错, 无法保存部门或用户

报错详情:

pymyagl.err.InternalError:(1271, "Illegal mix of collations for operation 'case')

原因:

  1. ldap同步的数据信息(用户/部门及其属性等), 可能存在特殊字符(非utf8) [概率较大, 需排查]
  2. 用户管理的库/表的 库/表的 COLLATE 不是 utf8_general_ci (show variables like "collation_database"; / show table status;)

相关issue:

@ganzaofang
Copy link

ldap3.core.exceptions.LDAPSessionTerminatedByServerError: session terminated by server

image
image
issue: cannatag/ldap3#513

@wklken
Copy link
Collaborator Author

wklken commented Jun 30, 2022

ldap3.core.exceptions.LDAPSessionTerminatedByServerError: session terminated by server

ldap3.core.exceptions.LDAPSessionTerminatedByServerError: session terminated by server

image image issue: cannatag/ldap3#513

这个报错是使用开源包 ldap3 调用请求 ldap服务器报的, 非业务代码逻辑问题; 大概率是ldap服务/网络导致的, 也可能是配置的调用参数未被授权;

建议搜索下关键字: https://www.google.com/search?q=ldap3.core.exceptions.LDAPSessionTerminatedByServerError%3A+session+terminated+by+server&rlz=1C5CHFA_enHK987HK987&oq=ldap3.core.exceptions.LDAPSessionTerminatedByServerError%3A+session+terminated+by+server&aqs=chrome..69i57j69i61j69i60l2.258j0j1&sourceid=chrome&ie=UTF-8

按照搜索结果逐一排查, 例如这种: cannatag/ldap3#513 (comment)

@wklken
Copy link
Collaborator Author

wklken commented Oct 13, 2022

LDAP 同步时,当用户名来源不是 cn 时,会导致部门关系拉取上移一级 #356

dc=a,dc=b,dc=local
   |- ou=Group
       |- ou=SubGroup
           |- uid=abcdefg

这种情况下, 配置拉取节点 ou=Gorup,dc=a,dc=b,dc=local

会发现, uid对应的用户, 绑在 Group 下, 而不是SubGroup

#356

这个问题尚未解决, 暂时无法支持到uid=abcdefg的这种ldap数据

@wklken
Copy link
Collaborator Author

wklken commented Oct 13, 2022

怎么清理某个目录下已同步的所有用户和部门

什么情况能清理:

  1. 测试环境, 确认数据不要了, 想要重置目录的同步配置
  2. 想要改拉取节点, 但是发现改了会同步失败Duplicate entry 'xxxxx' for key 'code'
  3. 新建了一个目录测试, 同步完, 想要建立一个正式的目录, 配置同一个ldap/mad源, 发现数据同步不进来

清理掉意味着废弃掉已有组织架构及用户, 如果权限中心已经同步组织架构过去了, 那么不能清理 (要么用户管理+权限中心都重置) => 如果清理, 权限中心下次同步将会出现数据错乱/权限错乱, 无法恢复


清理sql

select * from categories_profilecategory; 
# 获取 category的id, 替换下面sql的x

# 删目录下用户
delete from profiles_profile where category_id=x;

# 删部门
UPDATE departments_department SET parent_id = NULL WHERE parent_id IS NOT NULL AND category_id=x;
delete from departments_department where category_id=x;

# 删部门-用户关系表
delete from departments_department_profiles where department_id not in (select id from departments_department);

# 删用户-leader关系表
delete from profiles_profile_leader where from_profile_id not in (select id from profiles_profile);

@wklken
Copy link
Collaborator Author

wklken commented Oct 19, 2022

openLDAP sizelimit默认配置会导致只能同步500账号到bk-user

openLDAP的sizelimit默认值为500,会导致只能同步500账号到bk-user, 超限后新用户无法同步。
openLDAP限制参考:https://www.openldap.org/doc/admin24/limits.html

临时解决办法:
修改openLDAP默认值,参考如下:
image

更优解决办法:
建议bk-user优化同步逻辑,按需增量同步(比如用户第一次登录时,触发该账号和上下级关联账号同步),避免全量同步LDAP用户和组织架构到bk-user。

cat > sizelimit.ldif
ldapmodify -Q -Y EXTERNAL -H ldapi:/// -f sizelimit.ldif modifying entry "cn=config"

相关issue:

@wklken
Copy link
Collaborator Author

wklken commented Dec 30, 2022

SQL 补全

# 查看目录所有配置及其更新时间
select b.key, a.value, a.enabled, a.category_id, a.meta_id, a.create_time, a.update_time from user_settings_setting a, user_settings_settingmeta b where a.meta_id=b.id
and a.category_id=4;

# 查用户/部门数据(査几条看看, 必要的话order by id desc看最新的)
select * from profiles_profile where category_id=4 limit 10;
select * from departments_department where category_id=4 limit 10;

# 查询某个category中用户/部门启用的数据
select enabled, count(*) from profile_profiles where category_id=4 group by enabled;
select enabled, count(*) from departments_department where category_id=4  group by enabled;

# 査部门-用户关系表中的数据
select count(*) from departments_department_profiles where department_id in (select id from departments_department where enabeld=1 and category_id=4);
select count(*) from departments_department_profiles where department_id in (select id from departments_department where enabeld=0 and category_id=4);

@wklken
Copy link
Collaborator Author

wklken commented Feb 10, 2023

case 示例:

1. #915

  1. 脏数据导致部门无法被同步进来, 数据库报唯一字段code冲突
  2. ldap 配置问题导致数据源拉不到用户
  3. 拉取用户名是中文字符 => 修改属性

@wklken
Copy link
Collaborator Author

wklken commented Feb 15, 2023

误删除ldap目录后该怎么恢复?

注意第三步恢复部门及用户, 可能恢复后有脏数据(之前被软删除的也被恢复了), 因为此时无法识别哪些是之前删除的, 哪些是删目录的时候删除的

删目录删除的=>其update_time应该是一致的或接近的, 可以多加一个条件update_time>xxxx防止恢复了脏数据

注意: 如果不确定, 就不要操作; 如果能回滚db, 会是一个更好的选择

恢复sql

select * from categories_profilecategory; 
# 找到被误删的记录 获取 category的id, 替换下面sql的x

# 1. 恢复目录
update categories_profilecategory set enabled=1, status="normal" where id=x;

# 2. 恢复配置
update user_settings_setting set enabled=1 where category_id = x;
# 确认配置状态是ok的
select b.key, a.value, a.enabled, a.category_id, a.meta_id from user_settings_setting a, user_settings_settingmeta b where a.meta_id=b.id
and a.category_id=x;

# 3. 恢复部门及用户
update departments_department set enabled=1 where category_id=x;
update profiles_profile set enabled=1, status='NORMAL' where category_id=x;

@wklken
Copy link
Collaborator Author

wklken commented Feb 21, 2023

数据源用户名变更导致的同步失败

现象:

  1. 一直同步失败
  2. 显示同步成功, 但是用户数据, 部门-用户关系数据并没有同步成功

日志:

INFO [2023-02-20 14:45:41] 265 disable_profiles_before_sync 29280 140551249217344 
Going to mark profiles(and relations)(all: 7955) in category<2> as deleted, skipping 0 profiles 

INFO [2023-02-20 14:45:41] 212 _sync 29280 140551249217344 
======== Going to bulk_create(count: 236) for Profile ========= 

INFO [2023-02-20 14:45:41] 225 _sync 29280 140551249217344 
======== Syncing part of Profile(1/1) current: 0 + 236 ========= 

WARNING [2023-02-20 14:45:42] 239 _sync 29280 140551249217344 
Profile bulk_create failed, count=236, extra_params={}, will try to sync one by one 

ERROR [2023-02-20 14:45:42] 252 _sync 29280 140551249217344 
Profile bulk_create: save one by one fail, item=71470-abcdefg-NORMAL, will not be updated, detail={} 
Traceback (most recent call last):
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.IntegrityError: (1062, "Duplicate entry 'f94236485e704bef717341c34d6c81c0d20cc7ad12c916b8cd9960c712db142e' for key 'code'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/bkce/usermgr/api/bkuser_core/common/db_sync.py", line 232, in _sync
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/query.py", line 503, in bulk_create
    objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts,
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/query.py", line 1293, in _batched_insert
    self._insert(item, fields=fields, using=self.db, ignore_conflicts=ignore_conflicts)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
django.db.utils.IntegrityError: (1062, "Duplicate entry 'f94236485e704bef717341c34d6c81c0d20cc7ad12c916b8cd9960c712db142e' for key 'code'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/bkce/usermgr/api/bkuser_core/common/db_sync.py", line 243, in _sync
  File "/data/bkce/usermgr/api/bkuser_core/profiles/models.py", line 166, in save
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/base.py", line 740, in save
    force_update=force_update, update_fields=update_fields)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/base.py", line 778, in save_base
    force_update, using, update_fields,
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/base.py", line 859, in _save_table
    forced_update)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/base.py", line 912, in _do_update
    return filtered._update(values) > 0
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/query.py", line 802, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 78, in _execute
    self.db.validate_no_broken_transaction()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 448, in validate_no_broken_transaction
    "An error occurred in the current transaction. You can't "
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.
ERROR [2023-02-20 14:45:42] 252 _sync 29280 140551249217344 
Profile bulk_create: save one by one fail, item=71471-aabbcc-NORMAL, will not be updated, detail={} 
Traceback (most recent call last):
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
pymysql.err.IntegrityError: (1062, "Duplicate entry 'f94236485e704bef717341c34d6c81c0d20cc7ad12c916b8cd9960c712db142e' for key 'code'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/data/bkce/usermgr/api/bkuser_core/common/db_sync.py", line 232, in _sync
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/query.py", line 503, in bulk_create
    objs_with_pk, fields, batch_size, ignore_conflicts=ignore_conflicts,
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/query.py", line 1293, in _batched_insert
    self._insert(item, fields=fields, using=self.db, ignore_conflicts=ignore_conflicts)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/cursors.py", line 148, in execute
    result = self._query(query)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/cursors.py", line 310, in _query
    conn.query(q)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 548, in query
    self._affected_rows = self._read_query_result(unbuffered=unbuffered)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 775, in _read_query_result
    result.read()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 1156, in read
    first_packet = self.connection._read_packet()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/connections.py", line 725, in _read_packet
    packet.raise_for_error()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/protocol.py", line 221, in raise_for_error
    err.raise_mysql_exception(self._data)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/pymysql/err.py", line 143, in raise_mysql_exception
    raise errorclass(errno, errval)
django.db.utils.IntegrityError: (1062, "Duplicate entry 'f94236485e704bef717341c34d6c81c0d20cc7ad12c916b8cd9960c712db142e' for key 'code'")

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/bkce/usermgr/api/bkuser_core/common/db_sync.py", line 243, in _sync
  File "/data/bkce/usermgr/api/bkuser_core/profiles/models.py", line 166, in save
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/base.py", line 740, in save
    force_update=force_update, update_fields=update_fields)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/base.py", line 778, in save_base
    force_update, using, update_fields,
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/base.py", line 859, in _save_table
    forced_update)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/base.py", line 912, in _do_update
    return filtered._update(values) > 0
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/query.py", line 802, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
    cursor = super().execute_sql(result_type)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
    cursor.execute(sql, params)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/utils.py", line 78, in _execute
    self.db.validate_no_broken_transaction()
  File "/data/bkce/.envs/usermgr-api/lib/python3.6/site-packages/django/db/backends/base/base.py", line 448, in validate_no_broken_transaction
    "An error occurred in the current transaction. You can't "
django.db.transaction.TransactionManagementError: An error occurred in the current transaction. You can't execute queries until the end of the 'atomic' block.

查看日志, 每一条用户插入都报错, 并且异常信息都是一致的 pymysql.err.IntegrityError: (1062, "Duplicate entry 'f94236485e704bef717341c34d6c81c0d20cc7ad12c916b8cd9960c712db142e' for key 'code'"); 这里冲突的code是同一个

排查:

select * from profiles_profile where code='f94236485e704bef717341c34d6c81c0d20cc7ad12c916b8cd9960c712db142e';

找到这个用户, 然后到ldap/mad数据源查找这个用户, 或者咨询人事/ldap或mad维护人员, 最近是否有变更过这个用户的用户名(用户管理ldap/mad 配置中映射到username的字段)


失败原因: ldap/mad 数据同步到用户管理后, username是域下面唯一的, 并且会使用code=sha256(category_id + dn)作为用户的唯一标识(数据库唯一性约束); 此时如果在数据源变更了某个用户用户名字段(ldap 的cn或者mad的sAMAccountName, 取决于配置用户管理目录同步配置的用户名字段映射到ldap/mad的字段), 再次点同步, 由于username变了, 但是dn并没有变, 会导致用户管理当成两个用户进行同步, 但是code唯一性冲突

修复: 确认对应账号的username改成什么之后, 在用户管理profiles_profile表, 需要同步把冲突用户的username更新为数据源中最新的值, 然后重新进行同步


拉取逻辑:

  1. 通过ldap接口获取所有用户列表, 以及每个用户属性
  2. 通过用户属性中的dn字段, 计算每个用户的code
  3. 执行批量插入db
  4. 此时由于用户列表中存在code重复 / 或者同存量数据code冲突导致批量插入失败 => 事务回滚

@nannan00
Copy link
Collaborator

nannan00 commented Jun 14, 2024

配置 MAD / LDAP 登录慢 或 超时

#1724

测试是否 MAD / LDAP 查询慢

# Note: LDAP 引入
from bkuser_core.categories.plugins.ldap import login

# Note: MAD 引入
# from bkuser_core.categories.plugins.mad import login

from bkuser_core.profiles.models import Profile
profile = Profile.objects.get(username="xxxx", domain="xxxxx")

l = login.LoginHandler()
l.check(profile=profile, password="xxxxxxx")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Module: LDAP/MAD ldap/mad related Module: plugin DataSource Plugin Sign: help wanted Extra attention is needed Type: documentation Improvements or additions to documentation Type: question Usage or problems
Projects
None yet
Development

No branches or pull requests

4 participants