Monitoring - We identified multiple FPC/pfe crashes in our Juniper MX routers without any reason or crash dump. After investigation, we found an Report ID PR1785182 and we will open an JTAC in order to verify further this issue.

From what we learn from PR1785182, we found the following issue, which is compatible with what happened in our router.

On Junos MX platforms with Modular Port Concentrators (MPCs) 7/8/9 line cards, MX10008/MX10016 platforms using LC1201 or LC480 line cards, SRX5K platforms with Services Processing Card (SPC3) cards, due to memory issue causing hardware issue, the Flexible PIC Concentrators (FPC) reboots silently without generating any crash files.

As workaround we applied:

In our case, we use LC / MX10008 platforms, we performed full reboot and now the router forwarding is stable.

We'll schedule firmware update from 21.4R3-S5.4 to 22.2R3-S3 in order to workaround these issues.

Juniper Problem Report ID: PR1785182

May 08, 2024 - 23:44 GMT-03:00
Investigating - We had an outage in our router in Rio de Janeiro, which affected our operation in Rio de Janeiro and Espirito Santo. We are currently investigating this issue. The router is back to working as expected in the Routing Engine backup.
Other locations were not affected.

May 08, 2024 - 22:31 GMT-03:00

About This Site

This is the system status for Huge Networks, both edge network and dashboard, API, network security and management tools.

Customer Dashboards Operational
90 days ago
100.0 % uptime
Today
Customer API Operational
90 days ago
100.0 % uptime
Today
Customer Management Console Operational
90 days ago
100.0 % uptime
Today
North America Platform Operational
Cloud Agent Operational
Threat Protection Operational
PCI Compliance Operational
Web Application Firewall Operational
Network Backbone Operational
EMEA Operational
90 days ago
100.0 % uptime
Today
Paris Operational
Frankfurt Operational
London Operational
Milan Operational
Tel-Aviv Operational
Dubai Operational
Amsterdam Operational
90 days ago
100.0 % uptime
Today
APAC Operational
Tokyo Operational
Hong Kong Operational
Singapore Operational
Mumbai Operational
North America Operational
Miami Operational
Dallas Operational
New York Operational
Los Angeles Operational
Ashburn Operational
South America Operational
São Paulo Operational
Espírito Santo Operational
Rio de Janeiro Operational
Fortaleza Operational
Subsea Cables Operational
Miami x São Paulo ? Operational
Miami x Rio de Janeiro Operational
Miami x Fortaleza Operational
Seabras-1 Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Past Incidents
May 10, 2024

No incidents reported today.

May 9, 2024

No incidents reported.

May 8, 2024
Resolved - BGP import policies with match action drop (or md-cli reject) and bgp-leak, can cause memory utilization to increase whenever a route matches the policy entry's criteria. Any changes to policy, or removing, re-adding an import policy with such an entry, which matches a route or set of routes will also cause memory utilization to increase. In some cases where there are a lot of routes and/ or lot of changes, the memory utilization may increase to a point where the CPM runs out of memory and cause system wide instability.

In a redundant system, a dual CPM SWO can help clear the memory utilization, but the same set of actions will start increasing utilization again. [453453-MI]

May 8, 23:38 GMT-03:00
Update - We are continuing to work on a fix for this issue with Nokia. Currently our environment is stable and the problem will not occur again, but, we already informed Nokia regards this issue. In the next 24 hours, we'll issue an RFO.
Mar 25, 01:59 GMT-03:00
Identified - The issue has been identified and a fix is being implemented.
Mar 25, 01:55 GMT-03:00
Investigating - We are currently investigating this issue.
Mar 25, 00:12 GMT-03:00
May 7, 2024

No incidents reported.

May 6, 2024

No incidents reported.

May 5, 2024

No incidents reported.

May 4, 2024

No incidents reported.

May 3, 2024

No incidents reported.

May 2, 2024

No incidents reported.

May 1, 2024

No incidents reported.

Apr 30, 2024

No incidents reported.

Apr 29, 2024

No incidents reported.

Apr 28, 2024

No incidents reported.

Apr 27, 2024

No incidents reported.

Apr 26, 2024

No incidents reported.