我也查了一下资料。其实很多美国人也分析不清，当然他们用得很清，语感很清。
总结一下，我觉得这里的would用法，与虚拟语气分不开，是从虚拟语气转过来的，简化来的。
比如 I would'n know his age without asking him.
How would I know his age without asking him?
这里就有更多虚拟语气的痕迹。（without）
如果只说简化版的I wouldn't know his age, 或 I wouldn't know.对不清楚这个语气的外国人来说，理解就会有偏差，不能体会到那种细微的差别 （这里是和I don't know 做比较）。
分析到这里，也正好和你说的吻合起来了。
即：I would know 还意味着我有知道的
（必然性）原因。
它隐含的原来的完整句应该是： I wouldn't know if I were not one of them. How would I know without being one of them?