AAA Deep Dive on Cisco Devices

I’ve been working on some AAA configuration lately and I went through some of my older templates and realized that I didn’t want to simply use them without verifying first if I still believed that this was the best way of configuring AAA. I started by reading some of the official docs but quickly realized they were a bit shallow and lacked any real detail of some different scenarios such as what happens when the AAA server is not available. I then realized that there also is a lack of blogs that dive into this into any detail. Being curious, I thought I would lab it out as I have recently built an ISE lab.

The goal of this post is to start with a very simple AAA configuration, expand on it, verify each step what happens when the AAA server is available and when it is not. I will give you relevant debug outputs as well as my thoughts on different parameters in the configuration. Buckle up! because this is going to be a super deep dive!

We start out by applying a simple AAA configuration, where I have specified my ISE server, which is at 192.168.128.20, and has the device admin role enabled as well as the needed configuration for the NAD. We only enable AAA for authentication at this stage. There is no authorization yet…

aaa new-model
!
tacacs server ISE-EU
 address ipv4 192.168.128.20
 key tacacs
!
aaa group server tacacs+ ISE-TACACS
 server name ISE-EU
!
aaa authentication login VTY group ISE-TACACS local
!
line vty 0 4
 login authentication VTY

I then SSH to the device and the VTY method list is used. I’m allowed to start a shell:

AAA/AUTHEN/LOGIN (0000000D): Pick method list 'VTY' 
%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: daniel] [Source: 10.254.255.2] [localport: 22]
SSH2 0: shell request
SSH2 0: shell message received
SSH2 0: starting shell for vty
AAA/AUTHOR (0000000D): Method list id=0 not configured. Skip author

However, no authorization for exec has been configured yet, which means that when logging in, only privilege level 1 is granted:

NAD>show privilege
Current privilege level is 1
NAD>ena
% Error in authentication.

Note that I’m not allowed to enable to a higher privilege level. There is no AAA configuration for enable yet. What happens if ISE is not available? This is simulated by applying ACL blocking traffic to ISE:

RADIUS-SW(config)#ip access-list extended BLOCK-ISE
RADIUS-SW(config-ext-nacl)#deny ip any host 192.168.128.20
RADIUS-SW(config-ext-nacl)#permit ip any any
RADIUS-SW(config-ext-nacl)#int vlan 10
RADIUS-SW(config-if)#ip access-group BLOCK-ISE in

Relevant debugs when using the fallback account:

SSH2 0: authentication successful for fallback
%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: fallback] [Source: 10.254.255.2] [localport: 22]
SSH2 0: starting shell for vty
AAA/AUTHOR (0000000F): Method list id=0 not configured. Skip author
AAA/AUTHOR: auth_need : user= 'fallback' ruser= 'NAD'rem_addr= '10.254.255.2' priv= 0 list= '' AUTHOR-TYPE= 'commands'
AAA: parse name=tty1 idb type=-1 tty=-1
AAA: name=tty1 flags=0x11 type=5 shelf=0 slot=0 adapter=0 port=1 channel=0
AAA/MEMORY: create_user (0x7FEB4844A930) user='fallback' ruser='NULL' ds0=0 port='tty1' rem_addr='10.254.255.2' authen_type=ASCII service=ENABLE priv=15 initial_task_id='0', vrf= (id=0)
AAA/AUTHEN/START (2952777119): port='tty1' list='' action=LOGIN service=ENABLE
AAA/AUTHEN/START (2952777119): non-console enable - default to enable password
AAA/AUTHEN/START (2952777119): Method=ENABLE
AAA/AUTHEN(2952777119): can't find any passwords
AAA/AUTHEN (2952777119): status = ERROR
AAA/AUTHEN/START (2952777119): no methods left to try
AAA/AUTHEN (2952777119): status = ERROR
AAA/AUTHEN/START (2952777119): failed to authenticate
AAA/AUTHOR: auth_need : user= 'fallback' ruser= 'NAD'rem_addr= '10.254.255.2' priv= 0 list= '' AUTHOR-TYPE= 'commands'

Still no luck enabling as AAA is not configured for enable:

NAD>ena
% Error in authentication.

ACL is removed and we add authorization for exec:

NAD(config)#aaa authorization exec VTY group ISE-TACACS
NAD(config)#line vty 0 4
NAD(config-line)#authorization exec VTY

This means I am now allowed to start a shell with privilege level 15 as ISE has authorized me and applied the correct shell profile:

NAD#show privilege
Current privilege level is 15

Relevant debugs:

%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: daniel] [Source: 10.254.255.2] [localport: 22]
AAA/AUTHOR (0x10): Pick method list 'VTY'
AAA/AUTHOR/EXEC(00000010): processing AV cmd=
AAA/AUTHOR/EXEC(00000010): processing AV priv-lvl=15
AAA/AUTHOR/EXEC(00000010): Authorization successful
AAA/AUTHOR: auth_need : user= 'daniel' ruser= 'NAD'rem_addr= '10.254.255.2' priv= 1 list= '' AUTHOR-TYPE= 'commands'

Now, let’s once again try the fallback scenario where ISE is not available. When trying to SSH and login using the fallback account, this message is logged:

% Authorization failed.

Even though we have a local fallback account, we can’t currently be authorized because the ISE server is not available. We could fix this by allowing local accounts to authorize:

NAD(config)#aaa authorization exec VTY group ISE-TACACS local

This now allows us to login with the priv 15 fallback account:

NAD#show priv
Current privilege level is 15

Relevant debugs:

%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: fallback] [Source: 10.254.255.2] [localport: 22]
AAA/AUTHOR (0x11): Pick method list 'VTY'
AAA/AUTHOR/EXEC(00000011): processing AV cmd=
AAA/AUTHOR/EXEC(00000011): processing AV priv-lvl=15
AAA/AUTHOR/EXEC(00000011): Authorization successful
AAA/AUTHOR: auth_need : user= 'fallback' ruser= 'NAD'rem_addr= '10.254.255.2' priv= 1 list= '' AUTHOR-TYPE= 'commands'

This also works when using a privilege 1 account:

NAD>show priv
Current privilege level is 1

Relevant debugs:

AAA/AUTHOR (0x12): Pick method list 'VTY'
AAA/AUTHOR/EXEC(00000012): processing AV cmd=
AAA/AUTHOR/EXEC(00000012): processing AV priv-lvl=1
AAA/AUTHOR/EXEC(00000012): Authorization successful

There is also another way of configuring this. Instead of using local in the authorization config for AAA, we could use “if-authenticated” which would authorize any user logging in that has been autenticated:

NAD(config)#aaa authorization exec VTY group ISE-TACACS if-authenticated 

The difference here is that the fallback account only gets a shell with privilege level 1:

NAD>show priv
Current privilege level is 1

Relevant debugs:

%SEC_LOGIN-5-LOGIN_SUCCESS: Login Success [user: fallback] [Source: 10.254.255.2] [localport: 22]
SSH2 0: starting shell for vty
AAA/AUTHOR (0x14): Pick method list 'VTY'
AAA/AUTHOR/EXEC(00000014): processing AV cmd=
AAA/AUTHOR/EXEC(00000014): Authorization successful

The user would then have to enable to get full privileges:

NAD>enable
Password: 
NAD#

If you prefer to have the fallback account get exec shell with priv 15 directly, use the first method, otherwise use “if-authenticated” instead.

The next part is to configure authorization for all commands:

NAD(config)#aaa authorization config-commands
NAD(config)#aaa authorization commands 1 VTY group ISE-TACACS
NAD(config)#aaa authorization commands 15 VTY group ISE-TACACS
NAD(config)#line vty 0 4
NAD(config-line)#authorization commands 1 VTY
NAD(config-line)#authorization commands 15 VTY

With this configuration, all commands are authorized and ISE is the only source of authorization. We test by logging in and entering configuration mode:

NAD#conf t
Enter configuration commands, one per line.  End with CNTL/Z.

This works fine. Relevant debugs:

tty1 AAA/AUTHOR/CMD (1953132848): Port='tty1' list='VTY' service=CMD
AAA/AUTHOR/CMD: tty1 (1953132848) user='daniel'
tty1 AAA/AUTHOR/CMD (1953132848): send AV service=shell
tty1 AAA/AUTHOR/CMD (1953132848): send AV cmd=configure
tty1 AAA/AUTHOR/CMD (1953132848): send AV cmd-arg=terminal
tty1 AAA/AUTHOR/CMD (1953132848): send AV cmd-arg=<cr>
tty1 AAA/AUTHOR/CMD(1953132848): found list "VTY"
tty1 AAA/AUTHOR/CMD (1953132848): Method=ISE-TACACS (tacacs+)
AAA/AUTHOR/TAC+: (1953132848): user=daniel
AAA/AUTHOR/TAC+: (1953132848): send AV service=shell
AAA/AUTHOR/TAC+: (1953132848): send AV cmd=configure
AAA/AUTHOR/TAC+: (1953132848): send AV cmd-arg=terminal
AAA/AUTHOR/TAC+: (1953132848): send AV cmd-arg=<cr>
TAC+: (1953132848): received author response status = PASS_ADD
AAA/AUTHOR (1953132848): Post authorization status = PASS_ADD

But now we block the access to ISE again. What will happen?

NAD(config)#int lo0
% Authorization failed.

We are not allowed to authorize commands since ISE is not available. Relevant debugs:

AAA/AUTHOR/CMD: tty1 (98036820) user='daniel'
tty1 AAA/AUTHOR/CMD (98036820): send AV service=shell
tty1 AAA/AUTHOR/CMD (98036820): send AV cmd=interface
tty1 AAA/AUTHOR/CMD (98036820): send AV cmd-arg=Loopback
tty1 AAA/AUTHOR/CMD (98036820): send AV cmd-arg=0
tty1 AAA/AUTHOR/CMD (98036820): send AV cmd-arg=<cr>
tty1 AAA/AUTHOR/CMD(98036820): found list "VTY"
tty1 AAA/AUTHOR/CMD (98036820): Method=ISE-TACACS (tacacs+)
AAA/AUTHOR/TAC+: (98036820): user=daniel
AAA/AUTHOR/TAC+: (98036820): send AV service=shell
AAA/AUTHOR/TAC+: (98036820): send AV cmd=interface
AAA/AUTHOR/TAC+: (98036820): send AV cmd-arg=Loopback
AAA/AUTHOR/TAC+: (98036820): send AV cmd-arg=0
AAA/AUTHOR/TAC+: (98036820): send AV cmd-arg=<cr>
AAA/AUTHOR (98036820): Post authorization status = ERROR
tty1 AAA/AUTHOR/CMD (98036820): Method=NOT_SET
tty1 AAA/AUTHOR/CMD (98036820): no methods left to try
AAA/AUTHOR (98036820): Post authorization status = ERROR

What if we login with the fallback account, though?

NAD>ena
Password: 
NAD#conf t
% Authorization failed.

No dice! We are still trying to authorize commands and ISE is our only source. Once again, there are several options to fix this.
We could use local as fallback for authorization or we can use “if-authenticated”. What’s the difference? Let’s try with local first:

NAD(config)#aaa authorization commands 1 VTY group ISE-TACACS local
NAD(config)#aaa authorization commands 15 VTY group ISE-TACACS local

Below is user logged in via ISE. Notice that authorization is failing when not reaching ISE:

NAD#
NAD#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
NAD(config)#int lo0
% Authorization failed.

We can configure things if we use the fallback account:

NAD>ena
Password: 
NAD#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
NAD(config)#int lo0
NAD(config-if)#

Relevant debugs:

AAA/AUTHOR: auth_need : user= 'fallback' ruser= 'NAD'rem_addr= '10.254.255.2' priv= 15 list= 'VTY' AUTHOR-TYPE= 'commands'
tty2 AAA/AUTHOR/CMD (3518986582): Port='tty2' list='VTY' service=CMD
AAA/AUTHOR/CMD: tty2 (3518986582) user='fallback'
tty2 AAA/AUTHOR/CMD (3518986582): send AV service=shell
tty2 AAA/AUTHOR/CMD (3518986582): send AV cmd=interface
tty2 AAA/AUTHOR/CMD (3518986582): send AV cmd-arg=Loopback
tty2 AAA/AUTHOR/CMD (3518986582): send AV cmd-arg=0
tty2 AAA/AUTHOR/CMD (3518986582): send AV cmd-arg=<cr>
tty2 AAA/AUTHOR/CMD(3518986582): found list "VTY"
tty2 AAA/AUTHOR/CMD (3518986582): Method=ISE-TACACS (tacacs+)
AAA/AUTHOR/TAC+: (3518986582): user=fallback
AAA/AUTHOR/TAC+: (3518986582): send AV service=shell
AAA/AUTHOR/TAC+: (3518986582): send AV cmd=interface
AAA/AUTHOR/TAC+: (3518986582): send AV cmd-arg=Loopback
AAA/AUTHOR/TAC+: (3518986582): send AV cmd-arg=0
AAA/AUTHOR/TAC+: (3518986582): send AV cmd-arg=<cr>
AAA/AUTHOR (3518986582): Post authorization status = ERROR
tty2 AAA/AUTHOR/CMD (3518986582): Method=LOCAL
AAA/AUTHOR (3518986582): Post authorization status = PASS_ADD

By adding local to “aaa authorization commands”, we can configure things using a local account but the already logged in sessions will not be able to configure when ISE is not reachable. Let’s try the other option with “if-authenticated”:

NAD(config)#aaa authorization commands 1 VTY group ISE-TACACS if-authenticated         
NAD(config)#aaa authorization commands 15 VTY group ISE-TACACS if-authenticated

The ISE user can now configure even if the session lost contact with ISE:

NAD#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
NAD(config)#int lo0
NAD(config-if)#

Relevant debugs:

AAA/AUTHOR/CMD: tty1 (1078024099) user='daniel'
tty1 AAA/AUTHOR/CMD (1078024099): send AV service=shell
tty1 AAA/AUTHOR/CMD (1078024099): send AV cmd=interface
tty1 AAA/AUTHOR/CMD (1078024099): send AV cmd-arg=Loopback
tty1 AAA/AUTHOR/CMD (1078024099): send AV cmd-arg=0
tty1 AAA/AUTHOR/CMD (1078024099): send AV cmd-arg=<cr>
tty1 AAA/AUTHOR/CMD(1078024099): found list "VTY"
tty1 AAA/AUTHOR/CMD (1078024099): Method=ISE-TACACS (tacacs+)
AAA/AUTHOR/TAC+: (1078024099): user=daniel
AAA/AUTHOR/TAC+: (1078024099): send AV service=shell
AAA/AUTHOR/TAC+: (1078024099): send AV cmd=interface
AAA/AUTHOR/TAC+: (1078024099): send AV cmd-arg=Loopback
AAA/AUTHOR/TAC+: (1078024099): send AV cmd-arg=0
AAA/AUTHOR/TAC+: (1078024099): send AV cmd-arg=<cr>
AAA/AUTHOR (1078024099): Post authorization status = ERROR
tty1 AAA/AUTHOR/CMD (1078024099): Method=IF_AUTHEN
AAA/AUTHOR (1078024099): Post authorization status = PASS_ADD

What happens when a local user logs in?

NAD>ena
Password: 
NAD#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
NAD(config)#int lo0
NAD(config-if)#

This works as well. Relevant debugs:

tty2 AAA/AUTHOR/CMD (437306642): Port='tty2' list='VTY' service=CMD
AAA/AUTHOR/CMD: tty2 (437306642) user='fallback'
tty2 AAA/AUTHOR/CMD (437306642): send AV service=shell
tty2 AAA/AUTHOR/CMD (437306642): send AV cmd=interface
tty2 AAA/AUTHOR/CMD (437306642): send AV cmd-arg=Loopback
tty2 AAA/AUTHOR/CMD (437306642): send AV cmd-arg=0
tty2 AAA/AUTHOR/CMD (437306642): send AV cmd-arg=<cr>
tty2 AAA/AUTHOR/CMD(437306642): found list "VTY"
tty2 AAA/AUTHOR/CMD (437306642): Method=ISE-TACACS (tacacs+)
AAA/AUTHOR/TAC+: (437306642): user=fallback
AAA/AUTHOR/TAC+: (437306642): send AV service=shell
AAA/AUTHOR/TAC+: (437306642): send AV cmd=interface
AAA/AUTHOR/TAC+: (437306642): send AV cmd-arg=Loopback
AAA/AUTHOR/TAC+: (437306642): send AV cmd-arg=0
AAA/AUTHOR/TAC+: (437306642): send AV cmd-arg=<cr>
AAA/AUTHOR (437306642): Post authorization status = ERROR
tty2 AAA/AUTHOR/CMD (437306642): Method=IF_AUTHEN
AAA/AUTHOR (437306642): Post authorization status = PASS_ADD

Notice that the same method, “IF_AUTHEN” was used.

We could be done here, but there’s one more thing I want to try. What happens if a user that has privilege level 15 but a command set restricting what commands are acceptable,
such as not being able to configure BGP, tries to configure BGP when ISE is not reachable. I have created a command set in ISE blocking BGP for the “daniel3” user. The first
test is when ISE is reachable:

NAD#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
NAD(config)#router bgp 65000
Command authorization failed.

The user is not allowed to configure BGP. Relevant debugs:

tty2 AAA/AUTHOR/CMD (811311383): Port='tty2' list='VTY' service=CMD
AAA/AUTHOR/CMD: tty2 (811311383) user='daniel3'
tty2 AAA/AUTHOR/CMD (811311383): send AV service=shell
tty2 AAA/AUTHOR/CMD (811311383): send AV cmd=router
tty2 AAA/AUTHOR/CMD (811311383): send AV cmd-arg=bgp
tty2 AAA/AUTHOR/CMD (811311383): send AV cmd-arg=65000
tty2 AAA/AUTHOR/CMD (811311383): send AV cmd-arg=<cr>
tty2 AAA/AUTHOR/CMD(811311383): found list "VTY"
tty2 AAA/AUTHOR/CMD (811311383): Method=ISE-TACACS (tacacs+)
AAA/AUTHOR/TAC+: (811311383): user=daniel3
AAA/AUTHOR/TAC+: (811311383): send AV service=shell
AAA/AUTHOR/TAC+: (811311383): send AV cmd=router
AAA/AUTHOR/TAC+: (811311383): send AV cmd-arg=bgp
AAA/AUTHOR/TAC+: (811311383): send AV cmd-arg=65000
AAA/AUTHOR/TAC+: (811311383): send AV cmd-arg=<cr>
TAC+: (811311383): received author response status = FAIL
AAA/AUTHOR (811311383): Post authorization status = FAIL

Then access to ISE is blocked. Let’s try again:

NAD(config)#
NAD(config)#router bgp 65000
NAD(config-router)#

Et voila! This now succeeded. Let’s look at the debugs:

tty2 AAA/AUTHOR/CMD (3711593796): Port='tty2' list='VTY' service=CMD
AAA/AUTHOR/CMD: tty2 (3711593796) user='daniel3'
tty2 AAA/AUTHOR/CMD (3711593796): send AV service=shell
tty2 AAA/AUTHOR/CMD (3711593796): send AV cmd=router
tty2 AAA/AUTHOR/CMD (3711593796): send AV cmd-arg=bgp
tty2 AAA/AUTHOR/CMD (3711593796): send AV cmd-arg=65000
tty2 AAA/AUTHOR/CMD (3711593796): send AV cmd-arg=<cr>
tty2 AAA/AUTHOR/CMD(3711593796): found list "VTY"
tty2 AAA/AUTHOR/CMD (3711593796): Method=ISE-TACACS (tacacs+)
AAA/AUTHOR/TAC+: (3711593796): user=daniel3
AAA/AUTHOR/TAC+: (3711593796): send AV service=shell
AAA/AUTHOR/TAC+: (3711593796): send AV cmd=router
AAA/AUTHOR/TAC+: (3711593796): send AV cmd-arg=bgp
AAA/AUTHOR/TAC+: (3711593796): send AV cmd-arg=65000
AAA/AUTHOR/TAC+: (3711593796): send AV cmd-arg=<cr>
AAA/AUTHOR (3711593796): Post authorization status = ERROR
tty2 AAA/AUTHOR/CMD (3711593796): Method=IF_AUTHEN
AAA/AUTHOR (3711593796): Post authorization status = PASS_ADD

Because the “if-authenticated” parameter was used, the command was authorized. This kind of makes sense as the server is not reachable to check if a specific command is authorized or not. This means that if you use command sets in ISE, when the ISE server is not reachable, users may be able to do things beyond what they normally can. I don’t see this as a big problem due to the following reasons, though:

• If a user has priv 15, they are normally trusted to do everything
• You should ideally have at least two TACACS+ servers, preferably in different regions
• Setting up RBAC with a local account can be complex so it’s not certain you’ll fare better authorizing a local account

This has been a deep dive and I hope you enjoyed it! To finish this post, I’ll leave you with some recommendations and things to consider:

• Be careful with your AAA configuration so that you don’t lock yourself out
• By using a custom method list instead of default, AAA configuration does not get applied until you apply it to the lines
• This also has the benefit of only applying to lines that YOU specify, as opposed to all lines, such as console
• ALWAYS have a secondary way of accessing the device
• TACACS is a critical service so treat it as such. Have at least two servers available, preferably in different regions
• Consider if you want to use “if-authenticated” to authorize exec shell and commands when your TACACS servers are not reachable

Thanks for reading!